Yes, there would be a risk. If you are trading on the decentralized exchange then why do you need to link your wallet to the exchange instead you keep it on the centralized exchange?
There are many decentralized exchanges like Idex, Coinswitch(exchange aggregator), Uniswap, Kyebrswap, Bancor, etc. A decentralized exchange will be a non-custodial exchange that will not have any control over the user's asset. You can directly trade on these exchanges without storing any assets with them. You will have complete control over your wallet and the keys.