Once the difficulty starts getting too crazy people will not mine as much
People are going to continue to mine at
much higher difficulty levels, even if that means botnets.
And as long as mining doesn't stop completely (a total stop to hashing will prevent a difficulty adjustment), the difficulty will adjust downwards to a level that the marginal miner is eking out a small profit.