Finding the Ledger shouldn't be much of a problem: the thief can try the PIN a few times, then it erases itself.
Note: I still don't have a hardware wallet, so haven't tested this by myself.
I can confirm that for Ledger Nano S: after 3 incorrect attempts for the PIN I had to recover the wallet from seed.
I'm not sure how many characterse long the PIN is, but the chance of a thief stealing your funds is quite small. The "wrench to the head scenario" where the thief forces you to share the PIN is probably more likely.
For Nano S it's 8 characters. I expect at least the same for Nano X. And you can make it 8 chars or you can make it shorter - that could also confuse the thieves.