We can't really have a discussion unless you give actual numbers. The longest time it can take is infinite, with 0 miners. Don't know if that really answers your question or not.

We're about to see a drop back down to 8000 diff or so in 6 hours, down from 18000. So we will likely see more miners coming back and the cycle continues. Notice however that we now have a new minimum amount of miners (compared to starting with 0 ) This is part of the algorithm. Each cycle the base should continue to grow.