I have also noticed the var diff settings I also start at 3 and then it adjusts. I use CGminer and use the balance option under pool management, I split my shares between 2 slush pool workers nad it keeps my diff down at 1.... I know that's cheating but only running at 4.5 gh with a higher diff variance kills me! esp short rounds when its running at diff 3 or 4, I may only get a few shares in a round..
One question regarding split to more workers:
Is it possible to split workers and lower difficulty if I'm using stratum proxy on one IP address?
Link bandwidth isn't problem for me.
EDIT: mining with cgminer on Linux - running multiple processes = workers isn't problem
Yes it is quite simple to run more workers. You simply run a process for every worker with one config file for every worker / process.
If have set rc file accordongly allowing automated transfer of miners on certain triggers and events.