Each peer must check that there are no duplicate plaintexts after decrypting. Otherwise attacks on unlinkability are possible.
Is that the same as checking that no two recipient addresses are the same, once the shuffle reaches the last participant, or is there more to it?
Multiple participants submitting the same recipient account would be a trivial attack to counteract, however there is no way to protect against a real sybil attack in which multiple participants, each submitting a different recipient address, are actually controlled by the same entity.
Independently of James' work, we are also working on implementing coin shuffling using your algorithm in the upcoming version of Nxt. The blame phase is really the complicated part to get right, and here we are taking the approach to disclose the one-time keys used by each participant, to find and penalize the rogue participant. When ready, we would certainly welcome you to have a look at our implementation too.