All you need is the backup phrase so that you can recover your wallet. Check the hardware wallet's website for the instructions of recovery, they have it. For example, Ledger Nano S, you'll see the instructions on "Lost device, PIN code or recovery phrase".
So if I need that why wouldn't I just go paper wallet?
If you're thinking that way, then you should've trust paper wallet in the first place. I feel like you're running away, you don't want anything because you're afraid. However, you'll run out of options if you're focusing on the negative side. Let's say you use a desktop wallet, you're afraid of virus and malware, then secure your device, run scans, don't click malicious links, always check the url of a website, check the apps you install, something like that.
If you'll be keeping your LTC for a long time, this is another way I could think of:
1. Put it in a desktop wallet/android wallet that lets you hold the private key/recovery phrase
2. Store the key/phrase in a safe place then don't open the wallet anymore in your device after receiving your coins (you might want to uninstall it as well)
3. Just open the wallet when you need to do a transaction.
4. If you want to check your balance, then just check it on block explorer. It will lower the risk of being hacked.
I hope you won't be saying what if you lost your recovery phrase or something, remember you are responsible of keeping it. If you want you do many copies and keep it in different places