If the hardware wallets failed, millions of people would also lose their life
savings and this would have a negative impact on the whole community and
the value of their coins.
Should we trust the people who are responsible for these wallets? Can
the technical people respond, on why we should trust them?
You can't do anything if hardware wallets failed. But you have to trust someone sometime. Even though hardware wallets were designed with good purpose to enalbe you safe and easy way to store your cryptocurrency, you can't blame the company for not seeing some flaw and/if hacker brakes in. If hardware wallets failed I'll just say I wouldn't like to be in the position of owners of those companies then. The thing that can be done is putting code open source, but then you have other thing where not many people will understand the code.
You simply have to trust somebody. If you are that much paranoid use paper wallets.