You can't just access a wallet if you know it's address. You need to know the private keys, which are private.

Once you have even one single outgoing transaction, your public key is NOT private.

Right, the public key is never supposed to be private. The person you quoted is referring to private keys.

Public key is private also on unspent address. This is extra security especially against poor wallet implementations.