Technical challenges I see:
1. Security. Your site has a massive target painted on it as it stores data that represents money.
2. DDoS. As can be seen by what has happened with MtGox multiple times exchanges are juicy DDoS targets for those who wish to manipulate the market.
3. Extortion. If it hasn't happened already it will soon. "Pay us X BTC or we take down your exchange via DDoS for a week."
4. Performance. If you're successful your site needs to be able to scale to handle high volumes of transactions and high TPS rates. Designing a real-time site like an exchange to be able to scale is not as simple a task as it might seem.
That's only half the equation though as you also face challenges on the business and financial regulation side of things:
1. BTC is in a bit of a gray area right now when it comes to government regulation. That *will* change as it gets bigger and attracts more attention. You will need to be able to deal with current and future regulations.
2. Money transmitter status and regulations. Developed economies (so-called "1st world countries") have complex and strict laws that regulate businesses that transfer money. As an exchange it is likely you will be determined (at some point) to be a "money transmitter" and have to put up some pretty substantial security bonds.
3. Banks and banking laws. Banks are nervous as hell about accounts that send and receive large quantities of money to/from large numbers of people. If you're GM or some other large company this is not a problem. If you're some upstart with no history it will set off all sorts of alarms in the bank. In the past this has resulted in many exchanges having their bank accounts closed by the banks they deal with. Banks are very concerned about money laundering laws and rightly so, HSBC was just fined $1.9 billion over it.
Those are the challenges that come to mind immediately.