Wrong. It is about security too. If you don't spend from an address, then your public key is safe behind SHA256. There is no risk to SHA256 from quantum computers as far as we know today.
Your public key will be revealed
oncewhen you broadcast a transaction spending
your coinsfrom that address, which means the quantum attacker can RBF and steal your funds even though you only used your address once.