The idea is valid, to the extent that since alt coins are nearly self regulating on the blockchain, why couldn't we not do the same thing with an exchange?
There are some conceptual and experimental p2p exchanges out there, with no fees.
They are for exchanging crypto coins and not USD.

The ultimate would be a multi-crypto wallet-based p2p exchange.