A hardware wallet doesn't (shouldn't) need to connect to any manufacturer servers in order to work.
Exactly, and it should be completely open-source. It might even work completely airgapped, but if the firmware used is closed-source, then you can't know if the company can access your funds. Flawed RNG or malicious code inside cryptographic libraries can grant them total access, and you can't prove anything.
I don't know Ledger, but I know they were never open-source to begin with. We should trust nobody who claims to be in favor of privacy and security without complete transparency. Period.