When mining on eligius, I had it fail over to ghash quite a few times (I am on 60/60 Mbit fiber, so not my Intenet connection).
Switched it over to GHash as primary, and I have not seen a single failover.

I am in Europe, and the Eligius server in the US. Does not seem to work out too nice. For GHash I use the UK server.

Make sure you use servers with good connectivity.