Well I think its mostly the bad name the bitcoin itself has because of the criminals that are using it, but that should be no risk.
The other thing I'm thinking is that people are to scared to invest in it because its still a risk you are taking.
The criminal make their way in any currency they find, it is true that it provide anonymity for the criminals but for that we do not have to be denied with such a beneficial currency, the states have to find legal methods for it.
Criminals are low life scum bags. They jump into anything if they see it's profitable.
The reason people are not using bitcoin is not because of them though. I tend to believe most of the people are not aware of bitcoin.