Nothing is ever 100% safe, even with your private keys hidden someone can physically attack you and torture you until you give them up. In some cases people feel safer with a 3rd party. Not everyone wants to be their own bank.
Even if you store up your assets on an exchange, you still have access to it, and in your scenario can be tortured to give it up. Only when you assets is on an exchange it can also be stolen without any negligence from your end. I would say it is much safer with you, and safely in a wallet, preferably a hardware wallet.