All you need is the backup phrase so that you can recover your wallet. Check the hardware wallet's website for the instructions of recovery, they have it. For example, Ledger Nano S, you'll see the instructions on "Lost device, PIN code or recovery phrase".
So if I need that why wouldn't I just go paper wallet? Then I can put what the hardware cost into more coin anyway.