You make a very good point here which is being missed by most posters.
To bring up a similar case of compromised email. Bitmain was hacked and thousands of customers emails home addresses and gear purchased was leaked.
I purchased 100 plus pieces of bitmain gear. Does that hack mean many people will come to my home?
Not likely. But if I owned a ledger I would not have all my coins on it any more.
Of course nobody is going to come after you because they wouldn't know where to look. You could be a retailer or a middle man for some mining farm owner. Most likely the gear wouldn't even be at your house, same as Bitcoins. Even if they had all the dates of purchase, in this business 6 months is a lot of time.
Thinking like that would make every celebrity or a successful CEO live in a bunker and have bodyguards patrolling the garden 24/7. All rich people would have to live like one of those cocaine bosses from South America.
I don't have all my coins on my Ledger, but I feel pretty safe knowing that the private keys are unaccessible. Probably the weakest point is the software from Ledger that you install on your PC to access the wallet.