Make sure something isn't eating up your bandwidth. I had trouble until I worked out that bitcoind was munching on mine (though I also turned off QOS which I haven't turned back on yet).
What have you done with Bitcoind?
I just stopped it and restarted it. Whoever was connected must have connected to someone else. Either it hasn't been an issue since or when it has been, I haven't noticed it.
Longer term, I need to turn my QOS back on and put p2pool up in the priorities and bitcoind down (probably down in the crawl category). Though it's possibly I'm going to upgrade my internet before then anyway.