I agree with what the above are saying.
I'd also like to ask what your motives are:
Why are you doing this?
What do you intend to achieve?
What are you doing that is new or different from any other exchange?
What makes you the ideal person to do this?
Don't get a VPS. Best option would be to have your own dedicated servers. But getting that setup, and then a sys admin to maintain it is going to add another $50k /year for it to be done properly and safely.
You'll need a proper security audit. That's going to cost a lot too.
You'll also need and accountant and/or legal team.
You could just go with a VPS and your $3k developer, but that wont be secure, your bank will most likely shut you down, and when (not if) you get hacked, liability will fall on you for not taking necessary precautions. Even if nobody takes legal action, you can consider your reputation destroyed.
Even if you handle just cryptocurrencies, I think it's only a matter of time before they start to become regulated (meaning you'll probably need to get other licenses and registrations depending on your jurisdiction.)
Please please PLEASE, unless you're going to do it properly (at least a high 5 figure budget) just don't do it. Last thing cryptocurrencies need is another exchange to fail and get hacked because you feel like setting up an exchange.