The only issue here is not security, but privacy.
You have now basically a KYC approved wallet.
I never thought about it that way but yeah, if you ever need the easiest way to swap your coins when you have the KYC there. They probably have it already and I remember the breach with Ledger before with all the customers accounts. Well, still not okay with privacy. I understand that.
Additionally you are probably paying a bit higher than in FTX exchnage because you are a making a market order (swap). You are not setting a price , but buying from the order book
I did compare it and of course I would like it better when I can set the amount of price that I want to. The process is just a lot faster rather than moving your coins to the exchange then withdrawing it. Takes a lot more time.