Ledger is a good hardware wallet.
It used to be. I still have it and I have a choice to find another one and these concerns really are valid. Although the recover feature is an opt-in, we don't like it to be like that.
I can advise you to seriously consider Blockstream's Jade hardware wallet as it has all the code clearly visible on Github and a very low cost of around $70 or so.
I'm torn between this and Trezor. I am using their blockstream green wallet software and also saw their other product which is the hardware wallet. But I think, there's more confidence in the usage of Trezor as it's the other hardware wallet that's next or now over than the Ledger hardware wallets.
While if you ask me which is the best software, my choice falls on Electrum.
Yeah, it's an open source and one of the best wallets but it's not a hardware wallet.