Difficulty only will limit the amount of bandwidth and server load as a result of your mining.  Usually what I do is use vardiff for an hour or two and then whatever the difficulty that it settles at is the fixed diff that I use.  You're looking to set it where you'll be sending roughly 10-15 shares per minute usually.

This difficulty will not affect your hashrate.