The pool hash rate dropped to an alarmingly low 20PH there for about 10 minutes.
The problem was that the passthru stopped accepting connections at all, and dropped everyone, but kept running along without a peep about there being a problem.
My alarms of course went crazy as the hash rate spiralled down, but it took a bit of time to realise it was the passthru (since it doesn't report anything at all when running - another TODO to fix in ckpool ...)
I restarted both the main pool and the passthru to rectify the problem.
All the nodes were mining OK - they total about 20PH - but of course everyone was failed over there when I restarted the main pool.
Restarting the passthru immediately fixed the problem and the hash rate is now back to 50PH
The outage started at 07:58 UTC and was resolved at 08:10 UTC, so 60% of miners should see at least a 12 minute gap in their mining on the pool for that time, 4 minutes after the block.
looks like we also somehow lost a good amount of users from just yesterday. right now its saying 797 users. I could of swore that was over 900 yesterday if im remembering right.
Yep it was saying just over 900 - but the hash rate is still around 50PHs so I guess it was a couple PH of rentals ... and I'm certainly never concerned about losing rentals ...
Meanwhile, I've resolved the problem with the passthru (written some code to report the client count) so next time the passthru screws up like this I won't be in the dark about it.
The passthru machine runs 2 passthrus, one for kano.is and a 2nd for kano.space (that only a few people ever connect to)
The 2nd one, kano.space, was ok during all this except of course for the (unneeded) pool restart that made it reconnect.