But as it gets more popular and a direct impact in their operations start to appear, it´s reasonable to think action would be taken, even making it illegal.
For now theres no need to fear of a sudden government ban because the chance of it happening is pretty low. Here where I live, bitcoin is considered as an e-asset and is legal to use unless you do something illegal and those poker sites took down by US and bitcoin are different. If one were to happen though, I still won't be afraid because no one can now that I use bitcoin unless I say it out loud.
You are right. In fact, this is the one of the big reasons why Bitcoin is getting to be popular. The government could not touch it unless you would be announcing it to them because you are out of your mind. Any government knows that it is virtually impossible to ban or totally control Bitcoin that is why they instead introduced some regulations to allow Bitcoin to operate under certain conditions...this is a sort of a give-take process.