Of course, the escrow will be involved because he/she is the one who holds the funds and executes the agreement of both or all parties.
That is why I said that I used the word escrow for lack of a better word.
In a court system, people sign contract outside of the court, without any judge being involved. If and only if there is a dispute, then one party will bring the contract to the court for a judge to make a ruling. So I guess that "judge" would be a more appropriate word for what I am talking about.
Say Alice and Bob enter a contract, create a multi-sig from which any 2-of-3 can spend the funds, i.e. Alice, Bob and Charlie, where Charlie is a judge.
Alice and Bob sign a contract (with pubKey) and agree that Charlie will be the arbiter if a conflict were to arise, then send the funds to the multi-sig. Meanwhile, as relevant, they sign their communication.
If everything goes well, Alice and Bob cooperatively move the funds as per the contract.
Only if there is a conflict, then either Alice or Bob can hire Charlie, provide him with the contract, the signature and any other relevant information for Charlie to make a decision. Charlie can then co-sign the transaction to enforce the contract.
Unlike with traditional escrow, Charlie does not need to know about even the existence of a contract, unless there actually is a conflict to resolve. Also, Charlie cannot steal the funds without the cooperation of Alice or Bob.
Each judge set his own rules, expertise, etc. For example, he may offer to rule on weather or not a transaction has happened on some blockchain, allowing cross-chain swap without having to implement atomic-swap programmatically. In case of hiring someone to write some code, a judge could specialize in verifying that the work has been done, or assessing how much of the work has been done.
It can easily get much more flexible, but I don't want to risk adding confusion.... at least yet.
I hope that helps clarifying.