Who are these many?

Pools have come close to if not gone over 50% in the past. And large groups of miners left them and went elsewhere. When pooled mining first became a thing there were only a couple of major pools and everyone else was still solo mining. It's not a real concern for people who understand that everything is always going to be in flux with this.

The real major concern, which is still not a real concern, would be who controls the miners that are mining at these pools.
If some government wanted to take over Bitcoin mining they could over a period of years acquire well over 50% of the hash rate. But if they took part of the hash rate and pointed it to one pool another part to another pool and so on, nobody would ever know about it until all the sudden they pointed it at their own pool. But due to the power requirements which are beyond immense and the cost of the hardware which is also beyond immense it's not a real concern for anybody who is not a paranoid nut job.

-Dave