^^ This ^^ is correct

It's a tad bit more elaborate tough...
If a block hasn't been found for 20 minutes, the diff indeed drops to 1.

Now, let's assume we've just had a diff adjustment at height x
at height x + 1980, the guy running a couple of ASIC's on the testnet decides to turn them off, block @ height x + 1981 isn't found for 20 minutes (diff = 1), block @ height x + 1982 idem...

Now, when we finally arrivate at x + 2016, the average time between 2 blocks > 10 min and the current diff is 1....
Sooooo... At the retarget, the diff will once again be 1 (the minimum), but it'll stay 1 for the next 2016 blocks.
After another 2016 blocks, the average time between 2 blocks is just mere seconds, so the diff is adjusted to 2.
After another 2016 blocks, the average time between 2 blocks is just mere seconds, so the diff is adjusted to 4.
After another 2016 blocks, the average time between 2 blocks is just mere seconds, so the diff is adjusted to 8.

Sooo, in this case, if the ASIC guy turned his ASIC's off @ height x + 1982, and turned them back on @ x + 2017, he'd be able to mine thousands and thousands of blocks in MINUTES... It would take a ton of retargets before the diff was adjusted to a height that would bring the average time between 2 blocks to ~10 minutes.

However, since he's the ASIC guy, he'd still be able to mine most of the blocks compared to a CPU miner.

Let's say the ASIC miner only has 5 Th of hashrate, while a cpu miner has 5 Mh => the ASIC guy produces a whopping 1000000 times more hashes per second than the CPU guy, so the odds of him finding a block at diff 1 is 1.000.000 times higher... Offcourse, at this point, things like other factors come into play (network speed, network hashrate, which nodes you're connected to, time needed to verify a block, IO ,...) but still...