All you need is the backup phrase so that you can recover your wallet. Check the hardware wallet's website for the instructions of recovery, they have it. For example, Ledger Nano S, you'll see the instructions on "Lost device, PIN code or recovery phrase".