Great point.. I think a decentralized exchange should be possible, I just don't know how the originating fiat will transfer between users without a bank though. We could mail it to each other using carrier pigeons 
the idea is that the system would facilitate and increase standard transactions among individuals, so there would be no way to know that a bank transfer was made to buy bitcoins, something which is very easy to spot and regulate if the account holder is MtGox instead.