I have also noticed the var diff settings   I also start at 3 and then it adjusts.  I use CGminer and use the balance option under pool management, I split my shares between 2 slush pool workers nad it keeps my diff down at 1.... I know that's cheating but only running at 4.5 gh with a higher diff variance kills me! esp short rounds when its running at diff 3 or 4, I may only get a few shares in a round..