I've been using a Ledger Nano for years but not being open source is a deal breaker for me.
Not to be harsh, but if closed-source is a deal-breaker, what have you been waiting for in all your years of using Ledger? The wallet was closed-source back then and still is now. The question is how has that affected your use of it? The real deal-breaker and reason to abandon Ledger is their upcoming Ledger Recover feature. I agree with that.
...the secure element in all their wallets are all close source.
Secure Elements are generally closed-source, in Ledger and elsewhere. Even Passport I think is using a closed-source SE, the same one used by other brands like Coldcard.
A HW device can remain safe if it is not assembled / created by the hands of such companies as Ledger or Trezor. I am inclined to believe that sooner or later any manufacturer of hardware wallets may come under pressure from states, and even ideological companies (supporters of the ideas of the cryptocommunity) will be forced to surrender. Therefore, I believe that the safest HW wallet will be the device that you assemble with your own hands. The BTC-community needs to look in this direction and develop such an opportunity, when even an ordinary user without special technical skills can assemble such a device as a designer. Assembled the physical part, installed the firmware, connected it to electrum and use it.
Decentralized hardware wallets must also be decentralized, otherwise all this decentralization becomes meaningless. Bitcoin in your wallet should not depend on HW manufacturers anyone.
There is nothing wrong with your post except most people aren't interested in assembling their own devices. And they shouldn't have to do that to interact with cryptocurrencies. If that becomes a norm, then we can call it a day. Crypto has no future in that scenario.