Sorry, I meant on the cpu miner I ran -t 1 as well, to test it. It still caused the slow down. So to confirm

So both the gpu miner and cpu miner running -t 1 causes the slowdown.