I think the biggest factor is ignorance and fear of the unknown. People fear what they don't understand and many are also weary of things that "seem too good to be true" and bitcoin is usually even touted as a can't lose investment or a money that drug dealers and terrorists like to use, so people are often weary. The mere mentioning of the 'darknet' or 'deepweb' is enough to scare a lot of people off and lets not forget that there was a time when 99% of the times it was mentioned in the media it was done so with negative connotations like its use in drug deals and darknet markets and that puts a lot of people off from even investigating it and seeing the brilliance of the underlying technology. If you ever encounter someone who is skeptical or cyncial then just explain to them the actual uses genuine of it and how fiat cash can just as easily be used for bad stuff and bitcoin is so much more than just 'anonymous' money.