If you never reuse an address, then this will not be a concern of yours ever.
It will affect them, though, one way or another. Yes, if they take the necessary precautions then their private key isn't going to ever be calculated that way by an attacker. However, the ones who do have exposed their public key and do not take those precautions
will affect them. The market will be disrupted once an attacker finds out the first ever rewarded addresses' private keys. Imagine being able to include a million of bitcoins into circulation.
Then, you need to ensure that the attacker can't make the calculations quickly. If they do and specifically faster than the time that takes your transaction to be confirmed, then they can even spend your own money.
Nonetheless, it's considered an improbable scenario, currently.