You are right. That is why we have to use decentralized wallets. Advantages of decentralized wallets over centralized ones is the security which it posses. The Decentralized wallets does not depend on any third party services. The entire control of the wallet remains in the hands of the users.
Some wallets which claim to be decentralized may not be truly decentralized, if you lost your private keys for instance, you can retrieve them back from the wallet meaning its somehow stored in the wallet. Although this mechanism might be helpful in most cases it bridges the idea of decentralization slightly. Well, far better off than wallets created by email and some which even require KYC to access buying/swapping functions.