Banks should be afraid of loosing business and their precious fees. (Unless they start somehow incorporating blockchain)
Governments should be afraid because they loose control over their own currency. (Unless they start "minting"/mining bitcoins)
People should be afraid because there is no customer service to call in crypto-currency. You get scammed or hacked and no one will save you.

Changes are coming...