A hardware wallet doesn't (shouldn't) need to connect to any manufacturer servers in order to work. You can use a Ledger with Electrum as well just as easily. You need to use Ledgers software in order to get coin-specific apps onto the device as well as firmware updates but once that's done there no need to ever touch their software again until it's time to update firmware.

The problem(s) here is that:

1. Their firmware is closed source so nobody has any way of knowing what the firmware is doing.

2. They've introduced a recovery service which places on your device code that makes it possible to extract your seed, shard it and have it sent through your computer to other custodial servers. This is being sold as a feature but is fraught with danger and is an absolutely horrid idea for dozens of reasons that have already been covered.

3. For all we know the actual code that enables this ability could've been rolled into previous firmware versions either as a placeholder or as a test. So it may technically already be there and there's literally no way of knowing because it's closed source. Saying "I just won't update then" isn't enough to be sure.

4. The idea that firmware can allow the secure element to reveal the seed really is a non-issue and is being used to obfuscate the issue by Ledger when they say "any hardware wallet can technically spit the seed out if firmware tells it to" - yes but they're not sending your keys through your PC which may or may not be malware infected, outward across the internet to 3rd parties which you now have to trust. However this did highlight the deep misunderstanding many people had about how hardware wallets actually work - this was due in part to most people being uneducated on the technicalities and also Ledger's fault for constantly using the wording that your seed could never leave the device. Now it's "oh well of course they can you should've known that but only if our firmware allows it which now it does"

5. Another point that has been stated to death but needs to be repeated is that the idea of "if you don't opt in what's the problem?!" is misleading because there is nothing permanent in the hardware that stops the need for a button press to allow key extraction from being removed in a firmware update. So technically - Ledger could force you into this through a firmware update and extract your keys without out you doing a single thing. The ability is there. Even if today you had to press buttons to allow the recovery service there's nothing that prevents them from removing that requirement in the future. Or perhaps it's already removed and the button press is just theatre to make you think it's necessary. You can't know because they're closed source. "Trust me bro!".