As my opinion it's safe. Atleast safer than centralised Exchanges. If you have full access of your private key and phrase then there is no problem. Just use some popular Dex don't put your phrase or private key in a new Dex. And one more thing beware of scams. There is many Dex sites which is not original they are phishing sites of some popular Dex and if you trapped there  that will be a real problem.