Lower than 120 (i.e. 60 and lower) makes it interfere with stuff, like HD video playback, games (on Windows with -f 120 I can mine even when gaming at about half speed) and more if it's really low. Considering the performance difference is less than 10% I keep -f high to not impact usability.
What If you start the miner without any switches and also, without any programs running? I know on my Ubuntu, even smallest applications were affecting the performance.