Why don't they just open-source their firmware, at least with a restrictive license if they are not comfortable with unlimited freedoms of MIT or GPL or similar. That is the quickest way to dispel any fears that there is a backdoor in Ledger source code. But of course, they won't do that, so it's best to avoid any kind of wallet - hardware or software - where its operation cannot be independently verified.
Perhaps there are things in the firmware they don't want anyone to know. They've been talking about making their code open source, but they're also lying about what "open source" actually means. In other words, they want to use the phrase Open Source while keeping some of their code closed... which just raises more questions about what they're hiding.
Ledger is dirty.