Got it! I was thinking that the golden rule "If you don't own the private keys to your coins, you don't own them." applies to all wallets.
It does but the Seed is the Alpha and Omega.
The saying 'not your keys, not your coins' applies to things like crypto exchanges and web wallets where you don't have access to your private keys but also not to a seed. When you set up your Ledger wallet you were shown your Seed words but when you register an account with an exchange you don't get that. They own your coins. You ask them for permission to use your coins. They can give you access to them but they can also request additional information, KYC, proof of origin or simply scam you and steal them.
So in case the exchange stops working you have no way to recover your money. That is why it is so important to know the private keys and even more important to have the correct seed.