An exchange with your seed phrase sounds like it could be scammy a lot of attacks could take place on that as have been listed above. Remote attacks from a rogue employee on your servers could also happen.
Exchanges are prone to being hacked because they're good targets. You've left out many important details in determining if your idea is secure or not: like where are the exchanges keys stored for sending funds, where are email addresses stored to prevent against phishing, where are IDs, usernames and emails stored to prevent against user doxxing, impersonation
Im suggesting no storage of private keys. So nobody has the private key - they are regenerated from the seed phrase which the user provides when they want to withdraw. Is this a terrible idea?
Regarding the other security issues, I work in online payments and most of those issues are solved by good practices, and AWS provides some good tools also.