One day, all these bitcoins will somehow hit the market. 
It's very possible that our coins are already mixed with illegal coins those were marked on the addresses. You never know. Does this worry you much?
It doesn't for me.
When we deal with cash, we do not know the source of that cash whether it came from an illegal or legal source one hand before me. I think the Bitcoin hatters are very active to plant the seeds in our head some day we will be in trouble. There will be time when we will not have these centralized exchanges with centralized fiat, there will be a time when we will not have any centralized financial system. Wait for those days.
PS: I am not encouraging those illegal Bitcoin traders by any means. I just think that nothing can stop Bitcoin.
Bitcoin is very actively used in the Dark Net. Therefore, there is no guarantee who had your coins in the past. It's the same as with paper dollars ... yesterday this money could be used to pay for drugs or weapons, and you got change with them in the supermarket. Because of this, you did not become a criminal.
Just like what Royse777 stated above. We have no idea where are our coins originated.
As long as you know you're not doing illegal activity, you are good to go and you don't have to worry.
Same with our fiat money, we don't know how many hands had it passed on.
And with crypto mixers, very likely that at some point we encountered some of those illegal bitcoins.