Bitcoin are stored in blockchain, if hardware wallets/non custodial wallets became unusable, it means the blockchain are no longer safe, if that happens, Bitcoin won't survive at all.

If you're mean about specific hardware wallets, it won't be enough to ruin Bitcoin because we still have an option to create DIY wallets.