Difficulty went down a little bit at the beginning of 2013, it scales up and down with available hashing power.  Hard to believe it could ever go down again but who knows.  Increase the difficulty too sharply and people will start to exit the mining game, which will lower the difficulty and bring people right back in.  It's a self-regulating system.

EDIT:  Just brainstorming, but could someone with a sizable percentage of the network's hash power manage to game the difficulty system?  Power up enough hashing power to total, say, 25% of the network, mine until difficulty is too high, shut down, wait for a drop, repeat?  Unless it's more profitable to run that 25% all the time vs playing with the difficulty number.