If you are concerned that the party involved is not neutral, then the argument is invalid because if you trust someone to escrow for you then logically you would trust that person with at least that much money in the event that you were trading directly with them. The reason for this is because there is the risk of the escrow running away with your money, just as with a direct trade, however when you trust someone to escrow for you (and the person you are dealing with is not the same as the escrow) then there is the possibility that the other party will try to scam you which would result in your money being tied up for some time while a dispute is resolved, having to incur a fee even though the trade did not go through and the potential for the person scamming you being able to fabricate sufficient evidence that the escrow provider should side with them (the last one is unlikely).
What does the bolded part have to do with being a neutral party? Trusting an escrow with an amount of money is different than trusting an escrow will be fair in the event of a dispute between the buyer and seller.
Let me answer your question with another question.
If I were to trade you my Bitcoin for your litecoin, how would I be able to resolve a dispute if you had sent me your litecoin directly that I would not be able to do if I was acting as escrow for my alt?
What is the difference? In both cases there is the exact same potential bias