A user shouldn't send anything to such an address unless it knows all participants have their public key in there.

In other words, ideally, user A needs to somehow communicate with user B and verify the keys are correct.

If you, as a mediator, have sole knowledge about the public keys of the other participants in a transaction you are supposed to mediate, you are not a mediator but a mere proxy which requires trust not to fuck up.