As most users mentioned, if you can't audit the source code/device (when it's open source) or they make the system closed source, you only can trust them.
In case the hardware wallet fail, user shouldn't lose their money if they made backup and they should able to restore it to another wallet since they usually use same standard (BIP 39).
There's no problem if you put it in secure places such as Safe and encrypt the private key with passphrase (BIP 38)

But still, how to remember the passphrase? Write it down and put into another Safe?