I think I had this wrong too. I thought the last block of the previous epoch was the first block of the new epoch. So, I was thinking that setting the time on the last block of the previous epoch to "current time" and then doing the same again on the current epoch (if the epoch was completed in less than 5040 minutes) would result in the difficulty increasing again by a factor of 4. If the last block of one epoch is not the first of the next, then I can see the opportunity for the epoch duration to be continuously manipulated by always jumping back in time for the first block of the epoch.
Bitcoin's difficulty is adjusted every 2016 blocks, which roughly corresponds to two weeks. The difficulty adjustment is based on the goal of keeping the block generation time around 10 minutes. If blocks are generated faster, the difficulty increases; if they spawn slower, the difficulty decreases.
The last epoch is determined by the last block of the previous epoch, not the first block of the new epoch. It is not possible to manipulate the duration of the epoch by jumping back in time to the first block of the epoch. Difficulty adjustment is based on the actual performance of the network and cannot be manipulated arbitrarily.