It seems that physical access to the device will mean that the password will be hacked and therefore your coins will be lost.
I look at it the same way as I look at oxygen tanks. They will help me breath and survive under water for a specific amount of time. After that, I will drown if I don't get to the surface in time. As soon as you notice that your hardware wallet is missing, take your recovery seed out from its hiding and transfer your coins somewhere else as soon as you get the opportunity to do it.
This has always been my approach. If I was to lose any device with bitcoin on it - hardware wallet, mobile wallet, encrypted cold storage, whatever - then I would be moving the coins to new wallets as soon as possible. A hardware wallet will buy you time to do this, but it shouldn't be seen as permanently infallible.
+1.
It's a good idea to even have such a wallet safely generated and prepared in case something like that happens one day. It will be quicker to just recover an already existing wallet than creating one from scratch and writing down its seed words when you don't know who might be playing around with your lost HW. Imagine being 10 minutes late because you had to make sure your seed is correct and doing all the other steps and verifications according to your personal needs.