Their firmware is completely closed source but as the CEO of Ledger said in that podcast, over time, they'll open more source of their code until they reach a level similar to Raspberry Pi.
"Opening more source" "over time" can mean anything and is something I'll believe when I see it. And even if they start opening more of their source code -- as long as parts of their code stays closed source there will always be insecurity.
It also doesn't fix past 'mistakes'. For instance, they could have spied on users for the last few years, patch it out and then open-source the firmware.
It is easy to see that if you used the firmware before it was fully open, there will always be a risk that some of your information is compromised (by Ledger or others).
I think it's okay if Hardware Wallet firmware remains closed source, at some point I even agree with that approach because on another hand, even if certain company has an open-source firmware, how can you be sure that they are actually using the open-source code? Is it possible to verify in case of hardware wallets? Maybe I lack technical knowledge here.
Yes, you can and
should.
A good hardware wallet manufacturer will actually advise and instruct its customers how to download the firmware, verify its integrity and flash it. It should also make sure to have
reproducible builds; this means being able to easily check that the firmware download matches the code.
It should also be easily possible to compile it yourself, alternatively.
The guys over at
WalletScrutiny check popular wallets from time to time to see whether their builds (firmware blobs / binaries) match the open-source code. In case someone cannot / doesn't want to do it themselves, and they trust them, that's a good resource to look at.