because problem is not in the server itself, i believe its connection. vardiff should decrase not only cpu/mem usage, [which is not the problem if anyone can get a full hashrate] , but also significantly decrase network traffic.
I have the same issue as you. I've been monitorin my miners and they'll drop to 50% speed every 2 minutes or so for a couple of seconds. Apart from the loss of hashrate, also thermally this is really bad for cards.