Wrong. It is about security too. If you don't spend from an address, then your public key is safe behind SHA256. There is no risk to SHA256 from quantum computers as far as we know today.
Your public key will be revealed when you broadcast a transaction spending from that address, which means the quantum attacker can RBF and steal your funds even though you only used your address once.
You assume a key can be derived within moments, which is not correct. Just because something can be computed, that does not mean that it can be done instantly.
It seems that in total 25% of all Bitcoins reside on addresses which are vulnerable to such quantum attacks, because their public key has already been published. A part is on old outputs like P2MS or P2PK where the public key was already published when the funds were mined or received. But most of these coins are vulnerable only because the addresses were re-used, and their owners don't know about the problem and/or are lazy and/or are thinking Bitcoin works like Ethereum and your address is your wallet (which is not the case).
Do you have any data sources for this? I'm interested to know in specifics like how many Bitcoin exactly are contained in the old outputs.