Who's time?  Time in a decentralized network isn't deterministic.

Say x minutes pass so some nodes lower difficulty and one finds a block however due to differences in clocks some nodes believe it has been < x minutes. 

HARD FORK TIME.  Some nodes believe the lower difficulty block is valid and some believe it isn't.  There is no resolution possible.