Let's assume we lose almost all the rest of the miners. What's the longest time that can pass until the next difficulty adjustment?
We can't really have a discussion unless you give actual numbers. The longest time it can take is infinite, with 0 miners.

Don't know if that really answers your question or not.
I'm sorry if "almost all" means 0 to you. Let me try some easier questions for you.
The FAQ states:
it can lower difficulty 400% in 12 hours
First, what does 400% lower difficulty mean given a current difficulty of 18,246?
Why hasn't the difficulty lowered in the last 13+ hours?