I've said it before and I'll say it again: the reputation.
The ponzi scammers scares out the newbies whom they have scammed. Those talkative newbies then will tell people as if it's a breaking news. This therefore gives bitcoin a bad reputation.
Although it's the newbie's fault and no one else's that he had fallen victim into one of those HYIP/ponzi scams.

As for me, now that I've started using bitcoin as one of the major currencies, I will not stop from using it. Simply because all the advantages it gives like anonymity and being tax-free. This is the privileges that we are not getting and will never get from fiat money.