Wrong. It is about security too. If you don't spend from an address, then your public key is safe behind SHA256. There is no risk to SHA256 from quantum computers as far as we know today.
Your public key will be revealed when you broadcast a transaction spending from that address, which means the quantum attacker can RBF and steal your funds even though you only used your address once.
You assume a key can be derived within moments, which is not correct. Just because something can be computed, that does not mean that it can be done instantly.