A ledger is by far more secure than say a keystore, however the challenge then becomes is the ledger itself physically secure. Do you keep it in a safe area, who knows that you have it and how do you physically secure it, is fire damage a risk
Ledger and Trezor are both very secure (in fact, I find using them both to have a similar experience). Trezor used to have an exploit but is is now resolved as far as I know. I cannot speak for a keystore device.
Keeping any of them physically secure AFTER they are initiated is not that important, they both have built in PIN protection methods (Ledger reinitiaites after a PIN has been wrongly entered 3 times, Trezor doubles the interval in-between PIN attempts).
The other two risks that I consider to be much greater are the following:
1. the robbery in which you are pressured and made to reveal your PIN (with a gun against your head or your child's head). Here it may be helpful that you may hide your funds under one PIN and some other smaller amount under another PIN
2. that your recovery 24 word backup is found and disclosed. I would advice physically securing that particular piece of information
with much greater care than the Ledger device itselfAnd the option 1 is not such a pure fantasy as one might think. There is a link to a real story:
I would advise to anybody to buy a ledger, true, but it is even more advisable to keep low profile about the crypto one has.