Yes, you have to confirm it on your Ledger hardware...
In theory. Like I keep saying... there's no way to prove it. There's no way to prove there isn't a backdoor in Ledger's code for access to a user's keys without confirming it on the user's device.
"There's no backdoor and I obviously can't prove it"
--btchip, Ledger owner & co-founder
Can't prove it?
Can't trust it.
Don't use it.