The same reasoning can also be applied to credit/debit cards, leather wallets, and other "containers" of money: people who have or use them in public are clearly not that poor. Yet, we rarely see people being attacked just after paying at the checkout of a grocery store.
Well, because paying with cards is the norm, but if we go the similar way, if every time you pay, even $20 amounts, you take a wad of bills out of your pocket for a total of $5k you become an easy target for thieves. It's only a matter of time before they try to mug you. Cards are different because they would have to hijack you, go to the ATM get you to put in the PIN and take money out.
Even worse, unlike banknotes or credit cards, a hardware wallet as such is almost useless and impossible to spend from because it is unlikely that a victim will provide a PIN code voluntarily.
In the face of a $5 wrench attack, it depends on your resistance to pain and anxiety. Most people would transfer their funds to the robbers right away after a couple of hard blows to the head.