Heat related probably. Your CPU is idle most of the time, so it's running cool. When you break out the hash, the CPU is running full speed and generates more heat. As it generates more heat, it will start to slow down slightly unless you have some kind of crazy fan setup or liquid cooling.

That's my guess anyway.