Definitely the human factor that is mitigating against this original plan of bitcoin P2P transactions, that assures privacy between two parties. And this is as a result of mistrust, because nobody wants to be scammed, therefore it necessitates the importance and emergence of the middleman in transactions. I think that as long as there is no technology on ground yet to guard against being scammed in P2P transactions, then the exchanges and other forms of third parties will still be invited as intermediaries in bitcoin transactions. I really look forward to the day when we won't need any centralized exchanges to transact bitcoin, that was made to be decentralized.
This is going to be difficult, the greater the degree of privacy the greater the chances people act on ways they would not if their information was exposed for everyone to see, so people will have to adapt, one way is the use of a centralized third party which mediates between the two parties trying to make a transaction, the other option is to simply accept the higher risk and move on, and as you may guess both options have advantages and disadvantages.