It shouldn't affect adoption at all. But it would be nice if there were much more legal merchants accepting bitcoin than illegal merchants. By that I don't mean the illegal merchants somehow go away, because they won't, I mean more and more legal merchants must use bitcoin as well.
A sizable percentage of bitcoin payments are to scams, giveaways and ponzis. There are also such things that take cash as payment but there are exponentially more legit merchants who take cash payments as well, and this is where bitcoin needs to get to.
Even now the criminal use if the third or the second thing that comes to minds of regular people when they hear about Bitcoin. Right now it's the most strongly being associated with volatility, and general population views it more as a risky investment rather than a currency that they would use for their day-to-day transactions because of that. Something needs to change with either the users or Bitcoin in order to see a larger adoption - maybe people have to rethink their values and the relationship they have with banks, or maybe Bitcoin has to be more stable and convenient. At this rate adoption rate is just too slow.