Who's time? Time in a decentralized network isn't deterministic.
Say x minutes pass so some nodes lower difficulty and one finds a block however due to differences in clocks some nodes believe it has been < x minutes.
HARD FORK TIME. Some nodes believe the lower difficulty block is valid and some believe it isn't. There is no resolution possible.
Bitcoin already has the concept of time. I think there is a 1440 minute time acceptance window.