Obviously one person seeing one public key isn't the end of all your privacy, but (extra quantum protection aside) is there any privacy and/or security disadvantage to your counterparty learning the public key of the address they're sending to?
No.
It's called a
public key for a reason. When using cryptography outside of bitcoin, your public key is generally distributed freely. As you point out, the public key of every address which has ever had an outgoing transaction is publicly known. If there was a security risk to revealing it, then millions of coins would have been stolen by now. The only theoretical security risk is if someone develops a way to reverse the hashing process and turn a public key back in to a private key, which is a potential for quantum computing but not for several decades yet.
There is no additional privacy risk. The public key which is revealed is the public key only for the single address which they are scanning anyway, and not your extended public key to all your address (sharing this extended public key would be a big privacy risk). They can't derive any of your other public keys or any of your other addresses from it. Any information they can gain from knowing that single public key (your balance and transactions from the associated address) they could similarly gain simply by knowing the address (which they will anyway since they are scanning it).