[ajford]

A few years back, I was part of a group in the early days of commissioning a piece of research equipment that consisted of many racks of FPGA and GPU computing equipment in a specially modified shipping container. This thing was installed in a desert area, and had to be cooled by a couple of AC units.

The issue was similar. On nights where the temperature dropped too close to the dew point for too long, the units would freeze over. However, at the time, there wasn't any temperature monitoring. So someone figured out how to monitor the die temp on the FPGAs without changing the running code. Took them a few days. By the time they finished, someone realized they could tie streamers to the AC vent, which could be seen in the remote video stream.

Anyways, the fix was to connect to the network, switch the AC unit to fan only for a couple of hours, then switch them back on. If I remember correctly, it was like this for about 6-8 months before they finally had someone replace the AC system with a more commercial unit that could handle the condensation.