Why was the majority of this not in a cold wallet?
This.
Based on the OP I assumed (incorrectly) that the attacker "only" got 100% of the hot wallet.
Even tho only a small majority of the coins are ever in use at any time
Yes. I realize this. I cannot undo it (believe me, I would if I could).
Wow... just wow.
I thought you were better than that.
I never store keys on a webserver for a project involving customer funds. If all monies belong to the site operator that's their business, but if there are customer accounts I refuse to write code for someone who isn't willing to put the keys on a separate, heavily locked down server (preferably with no public ip).