Anybody who says you have to use the buttons to confirm actions is assuming that to be true. Since Ledger's code is closed, no one but Ledger knows for sure what their code actually does. Even Ledger admitted they can't prove their code doesn't have any backdoors. They lied, saying "...because you can't disprove a negative," but that's nonsense. Ledger can't prove their code doesn't have backdoors because Ledger's code isn't open.
Why don't they just open-source their firmware, at least with a restrictive license if they are not comfortable with unlimited freedoms of MIT or GPL or similar. That is the quickest way to dispel any fears that there is a backdoor in Ledger source code. But of course, they won't do that, so it's best to avoid any kind of wallet - hardware or software - where its operation cannot be independently verified.