Quote from: jtoomim on November 11, 2014, 03:18:19 PM

It is now looking like the issues we've been seeing in our datacenter with the potential ASIC-damaging bug was not related to version 2.5.7 per se, but is instead temperature-related, and we only noticed it now because the temperatures in our DC only recently started getting really close to 0°C. Version 2.5.13 does not appear to fix these issues. Zvi is looking into it further.

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Quote from: Biodom on November 11, 2014, 03:24:48 PM

Interesting. Did you try NOT to evacuate the heat that miners produce, but instead direct large fans on a stack of miners, so heat is distributed in the warehouse?
This might be a naive question because you certainly have a large density of miners. I am curious because one of my miners is hosted in WA.

This is what we're doing.

I designed our system with the intent of using 100% fresh air -- air would go through our miners once then be exhausted outside. With the revelation that SP30s have problems below about 8°C, we've had to revise that system, as it's currently -4°C outside, with a forecast low for tonight of -11°C. Over the last few days, we have set up a system whereby we can control the amount of fresh air vs. recirculated air we get by turning on and off fans. It's a bit rough still, and produces hotspots when we recirculate too heavily, but it works.

The tricky thing is getting the air to mix where you want it to mix. Our hot air containment system was designed with the intent of having the interior be hot and low pressure. In order to get air to recirculate, we have to make it hot and high pressure. This means we get hot air leaking out at every hole or crack that we have in the containment. Unfortunately, the most common cracks and holes we have in the system are the rack mounting bars for the SP30s, which means we tend to get uneven air temperature within a single SP30. Air on the side tends to go through the SP30, then circle around and come back out the front by the mounting bars, and then go through the SP30 again. On the sides of the machine, the air will make several circuits through the SP30, but in the center, the air goes through only once. We're working on sealing off those little cracks. There are other problems too, like making sure that the eight corners of our pod all are exposed to roughly the same temperature, and making sure that our intake and exhaust air streams get mixed well.

Quote from: Biffa on November 11, 2014, 11:08:25 PM

Is there a process to roll back?

You can click on "Firmware Upgrade" then "Manual Selection" to access several firmware versions which Zvi currently thinks are decent. You can download all of the old firmware at http://storage.googleapis.com/spond_firmware/spon_2.X.YY.tar, where X and YY are the version numbers of the firmware you want to use.