Wrong. It is about security too. If you don't spend from an address, then your public key is safe behind SHA256. There is no risk to SHA256 from quantum computers as far as we know today.
Your public key will be revealed once broadcast a transaction spending your coins, which means the quantum attacker can RBF and steal your funds even though you only used your address once.