I was already thinking about a site like that, but a completely different system

My idea: Something like an exchange
1) Both sides will create a profile
2) They will choose the person they want to trade with
3) They will choose the currency (each side another)
4) They will input their wallet address
5) They will send the amount of coins requested by the other side to a address proposed by the site
6) In meanwhile, the other side is doing the same thing
7) When both transaction person=>site are accepted by the site, a big green button will appear : "Are you sure do exchange XLTC for YBTC with PERSON_Z?"

If both sides accept this, the money will be send automatically to provided addresses (transaction "site=>person")
9) There will always be a big red button "stop the trade" and both sites will be refunded
Unfortunately I don't have time to make it work right now, so feel free to use my idea
I think this is an excellent idea; the exchange could make a little money by charging 0.25% to each party.
To the OP: if you have the required skills this is the way to go, eliminating the need for human escrow