No. It's more like you are saying that every building is doomed because you just realized that it might happen that firefighter crews are busy at two other building which are on fire at the same time, and it happens that the remaining crew gets stuck because of a mechanical breakdown of the truck en route to the site. That is insane. My argument was not insane.

Why it could not happen: for the same reasons it hasn't happened up to this point, even though pools were hacked, DDoSed to death, huge farms of GPU and FPGA and now ASIC miners came online. It hasn't happened because in reality bitcoin mining is much more resilient, and network does not come down "like a house of cards" when a major miner goes offline.
Sure, if Bitcoin continues to grow, mining will inevitably become a specialized (but not necessarily centralized) industry. There will be no more hobby miners, much like most of radio today is a regulated big business. But we are not there yet. There will be proprietary, closed-source ASIC miners competing with each others, there will be large co-operative companies where you or I can join, there will be Bitcoin businesses who also mine simply because they have vested interest in securing the network, there will be a rush to cheap electricity once technological bottlenecks are reached, there will be technological breakthroughs, there will be ASIC heaters in cold regions...

It's good that you are worrying, but man you are worrying about the wrong things.