I'm a GPU miner with about 600MH/s on one machine under one worker.  I haven't seen my Diff go higher than 1 during this whole testing phase, so it doesn't appear to be affecting my variance at all.