Why does the history matter? We all receive cash that has been used to buy or sell cocaine, we know it. So what? For once more, coin history a privacy problem, not a fungibility.
Let’s say you want to buy some bitcoins but you can’t buy it from an exchange for some reason and I happen to own some coins for sale. So are you telling me, it wouldn’t make any difference to you if I wanted to sell you some coins with very bad history? (Maybe Isis used my coins to fund their operations or I got my coins from an exchange hack and I am that hacker) I want to sell you my coins directly to you, without using any third party exchange/mixer. Would you take this deal?
According to your logic, you should.
If you say “no”, now you see why it matters.