If the adjustment back down worked so fast how do you regulate the amount of blocks found per 24 hour period? Should be 720 Blocks per day right? When some idiot uses nicehash and runs difficulty up they hit blocks well to fast and once they stop the network needs to change difficulty. In Theory If some one nice hashed in a few hours all 720 blocks then the network should not produce more blocks for next 20 hours or so. If it does readjust in like 2 hours then you would produce more than 720 blocks per day?
The network needs to and does change difficulty much earlier than "once he stops". He therefore can't mine 720 blocks within a few hours. Especially not since the V2 fork which implemented a more aggressive difficulty adjustment. If we would wait until "he stops", he would never stop and mine all 18.5M coins in no time.