You could hardly be more wrong. There is no conceivably scenario where difficulty drops quickly.

If the last 2016 blocks took less than 2 weeks, then difficulty goes up, not down.

The only way for difficulty to drop is for the last 2016 blocks to have taken more than 2 weeks.
And the most it can drop is by a factor of 4. But that means that the last 2016 blocks took at least 2 months.
So by definition a large drop take a very long time.

If all mining stops, it takes forever to mine another block.
If nearly all mining stops, then it will take many months to adjust difficulty.