Sorry, I meant on the cpu miner I ran -t 1 as well, to test it. It still caused the slow down. So to confirm
So both the gpu miner and cpu miner running -t 1 causes the slowdown.
It seams that on your system it's just one or the other,
but I wouldn't generalise it. It probably depends on the actual cpu.