I had an interesting conversation today among the other senior software engineers where I work.  We talked about exactly this topic.

I'm interested in your opinion too.  What exactly would you do to avoid what we saw today.   There is no doubt that the infrastructure at gox is failing-by-design, so I'd love to hear your ideas too.

Lets start with the event broker.  The lag is an obvious clue that its doing more work that it really has to.   What aspects of the design of this sort of system would you say are the easiest to change, the "lowest-hanging-fruit" which is probably at the core of the problem.

Even now that things are settling down, we see lag go from zero (idle) to over 20 seconds every few minutes.  This is a big clue to the issues, I think.

--D