Because if doing so is possible, then another kind of attack is possible. For example: an adversary miner could start producing blocks with future timestamps to drop difficulty, then jump back in time (if it is possible and if that does not require dealing with endlessly growing difficulty), and then create a deep reorganization, because a lot of blocks with smaller difficulty could create a sum of chainwork greater than the honest chain. That attack could be of course noticed by anyone, but if it matches consensus rules, then it will be accepted by existing clients. Also, there are some altcoins with different difficulty adjustment algorithms, if the same way of checking time is repeated in other coins, then that altcoins could be potentially easier attacked in this way, because their difficulty can be changed sooner than for example every two weeks.