If they are open source, can't developers verify that the Ledger has a chance to steal users' private keys? Did nobody try it? Now, I am curious about it. Do you remember the recent hack of Atomic Wallet, where thousands of users reported that their wallet was hacked and they did not use any phishing? If users did not use phishing, how was their wallet hacked? These wallets are not non-custodial anymore.
Ledger is not open source and it wasn't possible to know it was a lie that your seed phrase cannot leave the secure element, that was until they launched the Ledger recovery service, then their lies were exposed as well as many other flaws in the Ledger hardware wallet. Self custodial doesn't automatically mean safe, you have to also make sure the wallet is open source and the code has been widely reviewed, Ledger isn't a recommended hardware wallet and if you have their device, you should switch to other good alternatives.