The fact that bitcoin is volatile, they're scared of investing in having a negative thought of price dropping.
People are also not aware of what's the technology behind bitcoin, blockchain. They aren't aware that blockchain was one of the powerful and modern technology that should we adopt right now.
From my experience with people I know and have met in this sphere, it's a combination of people not fully believing in the long term viability of the coins and or the stupid comments of things like "it's only used by criminals blah blah" that makes them not want to stay in it for the long haul. The media hasn't helped in pushing certain negative stories or making it seem like it's this spooky thing the average person should be wary of, as opposed to promoting it as a very viable source of wealth and a way to change the world.
At first, I thought bitcoin was only used in illegal transaction but then I tried to study about it and I didn't regret me entering this crypto world, it has so many knowledge.