because problem is not in the server itself, i believe its connection. vardiff should decrase not only cpu/mem usage, [which is not the problem if anyone can get a full hashrate] , but also significantly decrase network traffic.
I have the same issue as you. I've been monitorin my miners and they'll drop to 50% speed every 2 minutes or so for a couple of seconds. Apart from the loss of hashrate, also thermally this is really bad for cards.
I have this problem as well, is there a fix?
I found that too, assumed it was intensity up too high or put the gpu on auto along with the fan in cgminer options g> c> a> f/g
edit : Also keeping it in a cold garage helps too;

yup, loosing 10% hashrate is not good , and thermaly killing gpu's neither.
couldn't find a fix, changed pools, made a comparison
https://bitcointalk.org/index.php?topic=153431.msg1628997#msg1628997MooC Tals; EricCartman; pintycar :
I can confirm a fix, after 2 hours of testing of new notroll backend :
-no more invalids! [no more extraonce and ntime errors]
-no more pauses for GPU-usage [no more thermal variance!]
-only a small portion of stales left @ acteptable 1,8%
congratulations on great upgrade Nicksasa!