Does you're ISP throttle P2P? If they do then bitcoin-qt downloading the bloackchain for the first time will look like you are downloading a large torrent file causing them to throttle the speed.

I've seen ISP's that throttle P2P down to 0.5 kpb/s before causing the program to time out.