If it gets stolen then they can't get access without knowing the credentials. Although you can still recover your wallet if you have made a back up of the seed which is provided when you first create a wallet on the Ledger.

Although you should probably still make sure you don't get it stolen