Quantum computer can reduce that to 80 bits.
And if that ever becomes the case, then bitcoin will move to quantum resistant signatures. Relying on then insecure hash functions and keeping your public keys secret is not a tenable solution.
You don't have to just "think" your public key is secret. You can make sure it is.
Keeping your public key secret means never spending your coins. As I said above, if your security relies on your public key being secret, then your security is broken. Long before this becomes an issue, bitcoin will fork to quantum resistant signatures.
If you think that 128 bits of security is insecure, then you should probably stop using bitcoin. Even if you believe that all your coins are protected by 256 bits of security, the many millions of bitcoin present in addresses with exposed public keys is enough to completely crash the price of bitcoin to zero if they were suddenly all stolen and everyone lost confidence in bitcoin's security.