The difficulty adjustment works well now when the hashrate is skyrocketing. It is too slow however to re-adjust when the hashrate suddenly drops, like from 1.1MH/s to 55kH/s.
This happened right now when the latest attack stopped at block 36086 with the difficulty at 131M. The next block was found only after exactly one hour.
You would expect the difficulty to drop significantly, but it went up to 144M.

The pools are actually the ones who see what is really happening since they do directly measure the actual hashrate the miners contribute to the network through them in every single second.
I could imagine a re-target method where the pools are communicating the actual hashrate that is pointing at them. This way the adjustment could be done as hashrate joins or leaves.
The current systems in use only react after the fact by looking at the timestamps of previous blocks, which is always too late and inaccurate anyway.
Already because faster or slower blocks due to natural variance will emulate a higher or lower hashrate, which means the adjustment will be fooled by hashrate increase or decrease which simply doesn't exist.
Of course that would open up also many attack vectors and it might be not viable at all.