Their firmware is completely closed source but as the CEO of Ledger said in that podcast, over time, they'll open more source of their code until they reach a level similar to Raspberry Pi.
"Opening more source" "over time" can mean anything and is something I'll believe when I see it. And even if they start opening more of their source code -- as long as parts of their code stays closed source there will always be insecurity.
Case in point, Ledger's software is already mostly open source, except for the firmware. And that's where the bodies were buried. So even if part of it gets open sourced, as long as some parts stay hidden, they will always have room for burying bodies. "Welcome to my basement officers, feel free to look around, just don't open the freezer, that one's off-limit."
I think it's okay if Hardware Wallet firmware remains closed source, at some point I even agree with that approach because on another hand, even if certain company has an open-source firmware, how can you be sure that they are actually using the open-source code? Is it possible to verify in case of hardware wallets? Maybe I lack technical knowledge here.
With Trezor you can download the source code and compile it yourself. Heck, if you feel especially nifty you can just go ahead and make your own Trezor clone [1]. Can't get much more trustless than that.
[1]
https://www.instructables.com/Making-My-Own-Trezor-Crypto-Hardware-Wallet/