When antminers turn off, difficulty lowers, wouldn't that vest time to mine is when everyone else is turning off there's?
Nope, wishful thinking. Thats the same thing people said when ASIC's were introduced. "When the GPUS' are turned off the difficulty will flatten out." That never happened. Just look at the hashrate historical record.
When antminers are turned off, faster and more powerful miners will be turned on. That is probably a more accurate statement. ;-)