Post
Topic
Board Hardware
Re: Novec 7000 Project [immersive evaporating cooling]
by
DeathAndTaxes
on 09/08/2013, 17:48:56 UTC
Hi OP, you could try to contact Allied Control (http://www.allied-control.com/index.php). I know they have built and run a 1T+ mining farm using over 6000+ spartan FPGA with immersion cooling. But they have dissolved the farm a few months ago because of the difficulty and the ASIC miner.

Yeah that is another option for working fluid.  IIRC they used Novec 649 (boiling point of 50C).
http://solutions.3mmagyar.hu/3MContentRetrievalAPI/BlobServlet?lmd=1351678101000&locale=hu_HU&assetType=MMM_Image&assetId=1319241050803&blobAttribute=ImageFile

It has a higher boiling point of 50C.   I think a fluid around 45C to 55C is probably the sweet spot for chip cooling. The lower the boiling point relative to ambient temp the more difficult it is to cool the gas to below the boiling point so it condenses back into a liquid.  The issue the OP had with pressure is that the heatsink/peltier were insufficient to extract heat.  The FPGA was injecting x watts into the working fluid and the heatsink was removing y watts.  y
The again you don't want too high of a boiling point because you start to run up against the thermal limits of ASICs and FPGAs.  If the working fluid reaches a max temp of 50C, then the surface of the chip package will be ~50C.  The internal die temp will higher.  How much higher depends on thing like the chip design, surface area, and chip package.  Using CPU and GPU as a data point they have thermal diodes as part of the die to record the internal temp and they run 10C to 20C higher than working fluid temp.  In a two phase system (liquid->gas->liquid cycle) the package surface is never going to cooler than the boiling point as the boiling is what "removes" the heat.    A different route would be using a much higher boiling point and some circulating pumps with a heat exchanger (radiator) and just keep the entire system in one phase (liquid).  The system would never boil and would use circulation to move the heat from the chip to the radiator, but where is the fun in that. Smiley  Anyone know what the upper safe limit is for various FPGA & ASIC designs?  If any have thermal throttling where does it kick in at? On edit: Looks like Avalons default is 60C and BFL is 65C.  Not sure if that can be pushed higher.

Semi-OT but my rant on the excessively high cost limiting what a lot of potential innovation
Novec 649 is still ~$300 a gallon though.  Actually I don't know of any working fluid that has a high Dielectric Strength that isn't $300 per gallon or more.  Pricing is also tightly controlled so no free market at work.  3M is the only major supplier, it holds a lot of patents which it aggresively defends (so chance of startup making a Novec-649 "compatible" fluid is essentially nil) and it doesn't allow discounting.  Distributors or retailers that get caught risk being blacklisted and with only one supplier that means you are out.    It is as shame 3M has never been able (or willing) to bring the cost down.  I know it is a highly profitable niche market and I am sure 3M margins are something like 90%+ but imagine how many more applications would be economical at $20 or even $50 a gallon.  Part of the problem is 3M doesn't really want there to be a cheap solution.  For example one commercial application is for the cooling of lasers.  It kinda is the same scenario as cooling a CPU/GPU/FGPA/ASIC but it is a lot more energy (like 1000W or more) in a much smaller space.  The laser isn't immersed the laser shell is filled with the fluid.  This means much less is used.  A typical laser might only use 250CC (~1/16th of a gallon).  The system is completely sealed so they only replacement fluid is what is lost during maintenance.  Trust me the stuff is treated like liquid gold but you usually need to top the laser back up (maybe 20CC or less).  Maybe someday another company without the vested interest to keep prices high will design and patent a much cheaper working fluid.  If they do the cynic in me says 3M will buy them out and jack the price up 1000%.