It is an issue the moment the person sends the coins that they received from you into an exchange. When they get asked what is the source of these coins, they will just point to the previous transactions, and now you are the next target on the line for whoever is investigating that. So it doesn't matter what you don't care about, and yes, we are innocent until proven guilty, but why would you want to expose yourself to that risk?. You should just make sure that any coins you send are not tainted while maintaining your privacy. The question is, how to even go about that.
Then you should always be wary of what you are doing; exchanges are not the only ones being compliant with Federal laws (if it is even illegal to use a mixer 
). I would think that asking for the origins of your Bitcoins would be rather tedious and makes things difficult for both parties.Realistically, no one actually cares about that. At any point of time, your coins are probably tainted in some form or another, with or without your knowledge. The whole issue with mixers being an issue is blown out of proportion, and frankly, it is the easiest way to maintain your privacy. Unless there is a feasible alternative of breaking the link without having to mix your coins in some form or another, that is probably your only option.
Let's face it, you won't get investigated by law enforcement for having tainted coins, or having used a mixer. If you want to be overcautious, then transfer the mixed coins around a few addresses and you can just tell them you got it from a face to face transaction with no receipts.
The method for plausible deniability has also been mentioned; use a HD wallet. You can only deduce that the addresses are somewhat linked, but with no concrete proof. If that floats your boat, then it would be another method.