Difficulty influence on reject shares, and reject shares is waste of computing power that can results in more block!

That's what really matter.

Rejects is waste of your GPU and electricity for nothing, you should minimize it.