Given the fact that the pool operators couldn't find any flaw in their billing system I think it could have been victim of a block solution withholding attack.
How would that cause the buggy balances we've been seeing?
I don't know. When I saw the buggy balances, I thought some hacker was able to manipulate the paid out sum to be able to do huge payouts to get as much as possible of the pool wallet. However neither did 50BTC post such transactions, nor have I been able find them. It rather looks like the pool wallet already was empty before the balances got corrupted.
Perhaps the system was not prepared for such an underflow of the pool wallet and did react in an unexpected way? Perhaps this happened when they tried to adapt the system to support DGM? I guess 50BTC needs to shed some more light on this. After all I do just look at the blockchain and take some guesses.