Isn't this true only if you know their respective public keys?
Yes, but public keys are supposed to be, well, public. If your security relies on keeping your public key secret, then your security is flawed.
As garlonicon points out, there are so many scenarios in which your public key is revealed that this should be assumed to be the default position (especially as people move to taproot addresses). In addition to the ones he has listed, consider all reused addresses and P2PK address. There are also many non-transaction or non-address related ways in which people expose their public keys, such as by using any non-Core wallet which sends xpubs to a server, or uploading their xpubs to some other service. Wallet software generally doesn't protect your public keys in the same way it does with your private keys, meaning you can leak them much more easily.
Thinking you are more secure because you think your public key is secret is a false sense of security.