It depends how far you want to take this line of thinking. Using a hardware wallet in public could certainly be seen as a potential advert to criminals that you hold a not insubstantial amount of bitcoin. So you only use your hardware wallet in private. However, when you bought that hardware wallet, you probably handed over your real name and address to the manufacturer. That information could very well have been sold, shared, or leaked, and so now your real name and address are linked to ownership of a hardware wallet. So maybe instead you bought the hardware wallet using a pseudonym, paying with anonymized bitcoin, and shipped to a PO box or other pick up location not directly linked to your real address. So far so good. But then all the bitcoin you have bought and sent to the hardware wallet were bought using fully KYCed accounts on centralized exchanges who we know sell your data to a huge number of third parties, not to mention being hacked for this data as well.

Not flashing your hardware wallet in public is probably a smart move, but you will never be able to completely mitigate against $5 wrench attacks unless all your use of bitcoin is completely anonymous, and almost nobody does this. It is wise, therefore, to have a system of plausible deniability set up just in case, such as the one I outlined above - additional passphrases which can be revealed to an attacker while your main stash remains safe behind different passphrases.