You can not eliminate the middle man in businesses, especially in this digital age, be it fiat or cryptocurrencies, because they have come to stay. Big exchanges like Binance, are cashing in on being the middle man in businesses, an unavoidable indirect business associates, in your face, waiting for you to go through them.
Certain cryptocurrencies such as bitcoin are able to eliminate middleman technically, whereas in this case it is a user's need to adjust to business preferences, especially because cryptocurrencies must really pay attention to the level of trust of the second party.
As innovation develops in the crypto industry, the role of third parties can also be built into code which is usually called a smartcontract, although some of these sometimes have serious vulnerabilities that cause greater losses.
Theoretically, you can eliminate the middleman; however, in practice, it's not as easy as it sounds. Even P2P transactions require some sort of middleman; how are you going to pay for the bitcoin you're purchasing? Unless we're talking about a physical transaction with cash in hand, which isn't something that can be deemed particularly safe, I don't know about others, but this whole decentralized movement sounds too inconvenient to me, especially if you're new to cryptocurrencies and trying to make your first purchase. For us, who already hold some coins, the transition is a lot easier.
Unfortunately, whether we like it or not, exchanges are convenient and are used by a large percentage of the cryprocurrency community.