Yes, its based on the collective time of the past 2016 blocks.

So if we all turned off (except 1 miner, because we need to get through those 2016 blocks), it would take....

Well, we need come values here.  Say the last miner was 1 Terahash.  The total network right now is ~300 Peta, so 300 000 tera, so we would be 300 000 times slower.

Instead of taking 14 days to find that 2016th block, it would be 4.2 million days, or 11.5 thousand years.

Then the difficulty would drop to ~137 000, and we could turn all our machines back on.  Once we turned back on that 300 Peta, it would take the reverse!  300 000 times faster to get through those 2016 blocks, 4.667*10^-5 days, or about 4 seconds.  So... whoever turns on "first" will just get every single block, or about 50 000 BTC.