The fact that governments could simply ban it, like China almost did. China officials even said that "legitimacy of our national currency can not be chalenged" or something along those lines, so there's one reason for governments to ban it. And governments banning Bitcoin would effectively push it back to underground where it's only used by hackers and drug dealers.
Also, the fact that there's no central authority regulating the price means that Bitcoin is not very suitable for exchange for goods and services, due to volatility. But that might? change when/if Bitcoin becomes bigger.