All that matters is it getting used, the rest doesnt matter.
If a significant portion of bitcoin related commerce is for illegal uses then anyone who is associated with bitcoin or that owns bitcoin would be somewhat associated with illegal activity (it would be implied they are somehow involved). If it is only a small percentage of the overall level of commerce then people who use and own bitcoin would likely not be implied to be associated with illegal activity