In addition, the very fact that you have a Ledger or Trezor indicates that you have a cryptocurrency. This information can be used by attackers.
You may very well already be advertising you likely own Bitcoin simply by: having a forum account, KYC'd at an online service, even Tweeted about Bitcoin.
A hardware wallet used properly already prevents wannabe attackers from accessing it online. I'd wager the number of people falling victim to Bitcoin theft from a hardware wallet wouldn't show up on a pie chart of ways people lost their Bitcoin to thieves.