Right, this seems pretty dead on. Try turning off your bitcoind (if you're on 0.8.1 anyway, which by your post I'm assuming that's your problem) when the lag seems pretty bad to see if it goes away like we did. We also realized that this started happening about the same time we upgraded to 0.8.1. Only thing is that your lag stops when you stop mining?
Well, turns out that mining had nothing to do with it...Just a coincidence that led me to believe so. Closing the 0.8.0 client makes my high latency go away almost instantly.
Thanks so much to Doc and farfie for bringing up this topic. The high latency was driving me nuts when gaming!