From my experience with people I know and have met in this sphere, it's a combination of people not fully believing in the long term viability of the coins and or the stupid comments of things like "it's only used by criminals blah blah" that makes them not want to stay in it for the long haul. The media hasn't helped in pushing certain negative stories or making it seem like it's this spooky thing the average person should be wary of, as opposed to promoting it as a very viable source of wealth and a way to change the world.
While you are right we should not depend on the media to try to make bitcoin popular and be used by more people around the world, after all we know that for the most part the media is bought by the government and they are not going to bite the hand that feeds them so they are going to spread all kind of lies about bitcoin if they have to, however mainstream media is suffering from a credibility crisis and more and more people are turning to alternative sources to try to find the information they look and that is good news for bitcoin.