I may have forgotten the specific details of the unlinkability (been away from that for some months) but afair the IP address can be associated with a total payment amount and the output addresses. The unlinkability only creates a new address for each payee for each payment, but doesn't hide this new address. Thus on the next spend of the change, the input to the ring it likely known. It is these sort of combinatorial attacks (other variations) that I think might breakdown Monero's anonymity. Smooth please do correct me if my recollection has failed me.