It mostly about he control over the macroeconomics which keep those people afraid about the increasing adoption of Bitcoin a and other altcoins. For instance, let us assume that a country has a lot of adoption of Bitcoin and the government wanted to change the number of FIAT in circulation, adjust the interest rates on credit, sell debt, etc. Most of those things would not have the effect they would want it to have because an important percentage of the population would be isolated from whatever the Central Bank or the Treasury of the Stated wanted to do with the National currency.
That alone is enough tho keep bankers up at night.
Your points are valid @Hispo but I think Bitcoin might not be able to hinder the government from engaging in its economic policies to control the economy because the currency has a limited supply and it is global which means that only a few citizens of a particular country might own it. Secondly, the government has not been able to control the influx of other currencies in my country. Currencies like dollars have always been in circulation side by side with our local currency. The government can also limit the circulation of their currency and adjust interest rates because Bitcoin will have little or no impact on their policy. I think their concern is mainly the failure of centralized exchanges and crypto-related firms, tax invasion, and control of the financial transaction of citizens.