No. VarDiff will in time will optimize your difficulty for your miners to maximize your submitted shares and minimize stales.

Some pools have no ability to do this and you have to work it out manually. Some allow you to set minimums. Slush does it automatically. Sit back and relax and let the pool do it's magic.

Hoppers check your stats, I see a strong "move along" signal.