Here is a little more detail about our watchdog function we built into ZhashOS.

When ever a GPU crashes, we log the details in our zhash.log file including "which" GPU had the problem and what it was.

This makes much easier to diagnose the problem GPU and address it precisely.

Here is one such entry in our log file to explain.


As you can see here, when a GPU makes multiple "problem" log entries within a 2 minute time frame, we know it is/has crashed and we take appropriate action to restart it.