Normally the mixer would know who was who and you would only have anonymity towards the public - but if you do the mixing only you will know where the money went.
Nothing about CoinJoin requires the party assembling the transaction to know the input/output mapping.
In the big splashy CoinJoin post I simplified it down a model where there was someone acting like a 'server' that did the join,
but I'd described it a year before in a more complex form that had perfect privacy and DOS resistance (though of course privacy is limited to the anonymity set size; though by building a CLOS network it can be arbitrarily large).
I simplified it because no one was picking up and running with the more complex and complete description-- doing so seems to have been a huge success, recent research suggests that coinjoins are now a pretty substantial chunk of the activity on the network.
Indeed you could use a non-private version with multiple stages such that at least one of them is completely trusted by each of the members, but I think the more complete protocols are a better way to get there.
No system for improved privacy is perfect, the important thing is just to make progress.
