Network adjusts difficulty on every block, which is targeted for 120 seconds. The ~4Mh/s that the new folks bring ramp up the difficulty about 8x compared to them not being around, which means if they were to leave once block times hit 120 seconds again, blocks would instead take 20 minutes. That would only happen for a couple of blocks - the network would readjust to 120 block difficulties after a couple of hours.