But how does one get their fiat in and out?
Building a decentralized exchange consists 99% of solving this problem, and 1% building an order book and web site. Of course everybody is focusing all their efforts on the 1% part of the problem because it's easier and more interesting to do it that way.
A viable plan for forming a distributed exchange would start with figuring out how to get 10000 Bitcoin users to open accounts and advertise on Localbitcoins. For price discovery they could just use a volume weighted average of all the applicable exchanges. This would work but no one is interested because it involves interacting with people in meatspace instead of writing new code.
I agree. I have solved this problem. I have figured out a way to make a P2P decentralized orderbook. I will post the solution in a few minutes here: