Nice and detailed write up, thanks for that.

I did something similar with SmartThings and smartplugs. Basically when AM notices any "service degradation", i.e. service is offline, device crapped out (i.e. # of devices less than expected) or "device is sick" (have no clue what that means - never saw that happened to a GPU) - then it triggers an action in webCore, which starts a 1 minute countdown timer. I get a notification on the phone with options to power cycle a rig right away or cancel. If no response is received within a minute then the rig is power cycled (if you are doing that to a GPU rig you need to ensure that "recover after power loss is turned on in BIOS).

One thing that didn't work for me is waits on the AM side - it kept sending reboot requests, so I basically created a latching switch, that allows a single reboot within a 5 minute interval - that allows plenty time for a rig to recover.

In all though, it's nice to see that we are thinking generally in the same direction