Do you think constant Firmware upgrades on hardware wallets are their weakness? I have gone through some firmware upgrades for some hardware wallets (Ledger) and I have to say for someone with good technical knowledge, it was not a good experience. 
If we are talking about Ledger, it was worse in the past, now it's easy-peasy. Everything is complete without the user having to disconnect the wallet from the USB cable. In the past, you had to press and hold the buttons, then let go of one button while you connect/disconnect. What can be done to improve the firmware upgrade for these devices ...to make it "Idiot Proof" ?
Talking about Ledger again. I would say they already are. It's just like an installation of any other software. A few clicks on the 'Yes' and 'Next' buttons and you are done. What happens when Ledger or any hardware manufacturer goes bankrupt and exploits are found and the developers are not there to plug the holes? (Export seed to Electrum?)
You can buy any other hardware wallet and recover your accounts from seed. If they use the same derivation paths for your coins, even better. If not, you might have to recover the seed in a software wallet to modify the derivation paths. You should of course secure your Bitcoin before you go meddling with software wallets for altcoins.