Indeed, this is also very important, especially to protect against DDoS attacks.
The current algorithm already tries to prevent this by not allowing a more than 4x factor adjustment every 2016 blocks.

I think we'd need a way to achieve the same end result when doing the continuous adjustments, but not a 4x factor for every block, obviously... it would need to prevent a 4x factor difficulty difference in the latest 2016 blocks compared to the calculated difficulty of the 2016 blocks before those.