I think the problem long term is that governments will try and destabilise it. They're scared of its potential for tax evasion and money laundering and will use their vast resources to destabilise to currency.
I don't think BitCoin is inherently drastically more prone to illegal use than say cash or commodities as a system of wealth transfer, although it is slightly simpler to use illegally. That being said, the old guard will undoubtedly take steps through various governmental or financial channels to attempt to stop BitCoin as it represents a threat to their existing economic model - wherein they control the economy.
As a community we can do some things to help stem the blood flow from this, and even heal the wound that causes it.
The first is to educate the people we introduce to BitCoin about the ways in which it is similar to cash in that it can be used illegally, but the majority of it's use is for legitimate purposes. Earlier is better on this one, sine the mainstream media has yet to really latch on to BitCoin as the next big scare for ratings. The better educated people are (or can easily become) when that happens, the more foolish the old guard will look for trying to badmouth it.