An exchange with your seed phrase sounds like it could be scammy a lot of attacks could take place on that as have been listed above. Remote attacks from a rogue employee on your servers could also happen.

Exchanges are prone to being hacked because they're good targets. You've left out many important details in determining if your idea is secure or not: like where are the exchanges keys stored for sending funds, where are email addresses stored to prevent against phishing, where are IDs, usernames and emails stored to prevent against user doxxing, impersonation