my fear is that this would be a complicated process to do and I may end up losing/damaging the USB/paper.
If you lost or damaged your hardware wallet, you still can recover your funds with your seed phrase that the wallet generates for you randomly. When you set up a wallet, write down your seed phrase on a paper and put in a safe place