The biggest worry has to be that there's a steep enough fall in the hash rate that it starts taking, say, an hour to solve a block (and potentially 4-6 weeks to adjust the difficulty). There's not much incentive for previously sidelined hash capacity to come online because the difficulty hasn't adjusted to increase the probability of you finding a block.
If the hashing power drops significantly enough to drag out the average block time above an hour, than not only does the odds of a particular miner catching the next block increase (even if it takes longer) but the odds of catching transactions due to backups of processing increases. So there is certainly an economic incentive for a miner on the sidelines to jump in if some major miners were to suddenly drop out, even before the difficulty is adjusted.
Hence, "not much".
Electricity cost would be the most logical reason for a miner being on the sidelines bly expected to come off the sidelines.
Don't assume that all miners have the same primary costs as yourself, or even most. Mining is literally free if you heat with electric anyway, and it's winter where you are.