The difficulty adjustment works well now when the hashrate is skyrocketing. It is too slow however to re-adjust when the hashrate suddenly drops, like from 1.1MH/s to 55kH/s.
This happened right now when the latest attack stopped at block 36086 with the difficulty at 131M. The next block was found only after exactly one hour.
You would expect the difficulty to drop significantly, but it went up to 144M.
The pools are actually the ones who see what is really happening since they do directly measure the actual hashrate the miners contribute to the network through them in every single second.
I could imagine a re-target method where the pools are communicating the actual hashrate that is pointing at them. This way the adjustment could be done as hashrate joins or leaves.
The current systems in use only react after the fact by looking at the timestamps of previous blocks, which is always too late and inaccurate anyway.
Already because faster or slower blocks due to natural variance will emulate a higher or lower hashrate, which means the adjustment will be fooled by hashrate increase or decrease which simply doesn't exist.
Of course that would open up also many attack vectors and it might be not viable at all.
If the adjustment back down worked so fast how do you regulate the amount of blocks found per 24 hour period? Should be 720 Blocks per day right? When some idiot uses nicehash and runs difficulty up they hit blocks well to fast and once they stop the network needs to change difficulty. In Theory If some one nice hashed in a few hours all 720 blocks then the network should not produce more blocks for next 20 hours or so. If it does readjust in like 2 hours then you would produce more than 720 blocks per day?