Quote from: JaredR26 on May 06, 2015, 10:41:54 AM

And since this guesswork isn't needed to draw the conclusions... why?

It helps with forensics to have multiple independent sources of information in order to cross-check facts. I first had an estimate of 55°C peak cold aisle temps based on my in-person observations. I later heard a report of ~57°C intake temps on an SP10 from another party who may have an axe to grind with ASICSPACE and might be lying. I then saw a screenshot of a KNCMiner device in their facility showing temps that would indicate intake temps around 50°C. Finally, I observed plastic deformation patterns that could be explained by hot aisle temps in the vicinity of 60°C-90°C.

Quote from: JaredR26 on May 06, 2015, 07:54:28 AM

It was hot. Too hot. Definitely less than 100C. Less than 70C. More than 40C. We can agree there for sure. Even if we could calculate them, we don't need more accurate numbers based on the conclusions you covered or the ones I reached...

Someone else might read this wondering how hot things need to be to damage an S5. I don't want people to think that they magically start to burn up as soon as the intake temperature exceeds 40°C. Accuracy is useful even when it's not necessary for the immediate task.

Quote

I wasn't saying that air pressure did the bend, air pressure only influenced the direction of bend at best.

I posit that air pressure, primarily, caused the runaway temperatures localized to a certain point on the boards.

I agree, mostly. I think that runaway temperature was mostly restricted to the hashboards and the intra-heatsink space. It's worth mentioning that all of these S5s were of the older variety that do not have any heatsinks between the hashboards and the side panels. With minimal surface area, very little heat is transferred from the hashboards to the air that passes through that space. Typically, I see a delta-T of about 1 or 2°C for that air when airflow is not obstructed. In the ASICSPACE case, that may have been 4x higher, but that would still only have been around 8°C. Furthermore, the pattern of deformation suggests that the deformation was greatest where airflow was highest, not where it was most stagnant. This implicates ambient temperatures more strongly as the dominant factor in the plastic deformation.

Quote

Specifically to this case... I looked up the coefficient of thermal expansion for PE and found from 80F to 135F you get ~0.5% expansion. Putting that in a right triangle gives you a bowing outwards of about 2.5cm from a ~17.5cm plastic strip. You said you measured 2cm at worst, so that's about right.

(Minus the expansion of steel and aluminum, which is much smaller.)

Most of the plastic deformation indicates a shear stress, not compressive stress. The only exception to this is the buckling that was observed in a small number of the most severely deformed plastic. The buckling of the plastic occurred entirely along the short axis of the plastic shields. If thermal expansion were the culprit, I would expect to see buckling along the long axis as well, especially along the line in between the front and back screws. That line is the least deformed part of the side panels. The bottom edge is more securely attached than the top edge as well, so if the main cause of deformation was thermal expansion, one would expect the bottom edge to show more buckling and deformation. The opposite is true. The greatest deformation occurred on edges which I expect had the greatest airflow (evenly along the top edge, plus the back and front edge near the holes for the tail exhaust and fan).

Much of the deformation occurred on the top edge, above the highest screw attachment point. That edge typically is relatively straight, but it sags outward and downward. Since this edge was not being squeezed, and if it were it wouldn't cause the edge to sag like that, I don't think thermal expansion is a satisfactory explanation. Thermal softening combined with airflow is.

The apparent buckling I observed on a few panels might actually have been fluid dynamic effects similar to ocean waves or sand dunes being created by wind rather than actual buckling.

Something just occurred to me. If the airflow was anterograde but slow through the whole intra-heatsink space, the air coming out the exhaust port would be very hot. The side spaces would not maintain their pressure as well, so airflow there would be more likely to turn retrograde. This would pull the hot exhaust air around 180° back in the side ports before flowing either out the top of the miner or through the gap in the panel between the screws. This could explain the pattern of deformation pretty well.

Quote

None of that even requires excessive intake or exhaust temperatures(whether present or not). That, combined with other things I know and the difficulty of calculating or approximating exhaust/intake temperatures, leave me stating 'The cause was localized cooling system failure on the miner caused by high incoming exhaust pressure, exacerbated by the fact that it was "too hot".'

I think high pressures alone might have been enough to cause the ASIC damage observed, but I doubt it. I think high temperatures alone might have been enough to cause both the ASIC damage and the plastic deformation, but I doubt it. I think the evidence is clear that both existed. Indeed, it's difficult to have a positively pressurized hot aisle without getting hot air recirculation and overheating everything. I don't think there's any reason to try to pin the blame on one factor.