Why would miners invest so much into electricity for half the reward? I see this ending badly
Absolutely not. That's how Bitcoin is designed. It tries to space out the block to once every roughly 10 minutes.
If there are less miners, difficulty would simply go down. It won't slow down anything, except for the last high-difficulty block. Blocks would still be generated on-schedule.