Bitcoin can certainly be used in a damaging way. It can be used to avoid taxes, as you've said, but most are concerned about money laundering and currency outflows. If done on a large enough scale, it can potentially do long term damage to an economy.