From my experience with people I know and have met in this sphere, it's a combination of people not fully believing in the long term viability of the coins and or the stupid comments of things like "it's only used by criminals blah blah" that makes them not want to stay in it for the long haul. The media hasn't helped in pushing certain negative stories or making it seem like it's this spooky thing the average person should be wary of, as opposed to promoting it as a very viable source of wealth and a way to change the world.
Yup, Bitcoin has this image of being used illegally, some people who aren't aware or doesn't have additional knowledge about btc tends to believe this and we can't fully blame them, btc has history of it being used in that way. That's why they are scared. And we know that btc isn't only about that, its just an event and history. Fiat currency can also be used illegally. However, people has this image of btc in their mind so it is our responsibility to spread knowledge and facts about btc so they can learn from us, and hopefully we can still change their idea of btc and cryptocurrency.