Hardware wallets are typically 2FA, meaning a PIN is required, so loss of the wallet shouldn't compromise your private key.