So if the server corrects with ideal diffs, then I could just ignore seting a minimum and let it settle in to that sweet spot on it's own after time, since for me when my hardware gets here will be 6 workers @ 100gh each. So I can either use one worker for the 6 identical workers at 71 or just forget setting a minimum and let them settle in on their own, assuming I didn't misunderstand what you just said.
Basically everyone can forget setting anything at all. The server will make things work.
But if you have a high hashrate like 600 GH then you may want to set a minimum difficulty, maybe 512, to make the ride smoother right after you connect to start mining. This avoids the server having to ramp up your difficulty quickly from 4. Instead you get a good difficulty right away.