Look at the way how Bitmarket works:

The (Bitcoin) seller sends his Bitcoins to the Bitmarket escrow service
The buyer directly wires the Bitcoins to the seller
The seller releases the coins in the escrow

So you don't have to deal with any real money at all.

Another nice thing is that you can operate such an exchange 100% anonymous over Tor - you just need to somehow gain the trust of the community.