It does when the hashing power of the network falls, which is basically when some miners turn off their machines for whatever reason. Historically, IIRC, this has only happened once: when the Bitcoin price fell to $2/btc a couple of years ago.
Is there a balancing point to the difficulty? At some point only the people who have millions to invest in mining will stand to gain anything from it. This is a good time for the governments and banks to start mining bitcoins so they can drive the difficulty up high enough so they can be in control of it.