Personally I mostly fear the following scenario, which anyone can do at home (if he has a billion dollars lying around):
1. Short lots of bitcoins.
2. Build a huge mining cluster and completely mess up the block chain.
3. Watch prices drop as panic ensues.
4. Profit.
5. Lather, rinse, repeat.
If it happends once there will be no secont time bitcoin will be dead.
I think Bitcoin can survive this happening once. The problem is if there is a reasonable expectation of it happening again. That's why the possibility of repeating it scares me.