The key issue here is that even if at this point you do need a physical button press to confirm/deny a Tx or seed sharding, there's is nothing inherent in the architecture of Ledgers hardware that restricts the device to operating this way forever. The required button presses are a firmware update away from not being needed at all. Which means that change could be made with or without your knowledge. "We promise we won't" Back to trust me bro.

Ledger keeps repeating that "all hardware wallets require trust" and people get lost in this because while on one hand it's true to some degree, not every wallet requires as much trust as one that's closed source which also has the ability via firmware to split and send seeds through your USB/Bluetooth connection, through your PC and then stored elsewhere.

"Oh but the shards are encrypted!" This only sounds good until you realize that Ledger themselves say that any device can restore the shards. So the encryption keys are either specific to ledger Hardware (meaning anybody with a Ledger has them) or they're stored at Ledger headquarters (meaning they have them and you have to hope they aren't leaked the way all those addresses and emails were). Any way you slice this it's frightening.