Its as if its safer to not put all your eggs in one basket with a hardware wallet.

You got it wrong, it's safer to use hardware wallet, all you have to do is keep your private key safe and you are good to go, even if the wallet get damaged, with your private key you still get access to your coins