It is an issue the moment the person sends the coins that they received from you into an exchange. When they get asked what is the source of these coins, they will just point to the previous transactions, and now you are the next target on the line for whoever is investigating that. So it doesn't matter what you don't care about, and yes, we are innocent until proven guilty, but why would you want to expose yourself to that risk?. You should just make sure that any coins you send are not tainted while maintaining your privacy. The question is, how to even go about that.
You just have to point the next on the line if you don't have anything to hide, but trying to hide for such reasons can raise suspicions. Exchanges don't really track smaller transactions but if they have been asked to provide the data then they will do it so, and if you want to avoid all these then you have to buy Bitcoin from exchanges then move to wallet and spend there from it and never receive bitcoins from anyone because it may have be tainted at somewhere since its origin but you care so much about that.
Privacy is actually different from the context of this thread, privacy is when you don't want to reveal how much money you have in your wallet.