It does when the hashing power of the network falls, which is basically when some miners turn off their machines for whatever reason. Historically, IIRC, this has only happened once: when the Bitcoin price fell to $2/btc a couple of years ago.
Is there a balancing point to the difficulty?
The difficulty reflects the average hashrate over the last 2016 blocks, and it aims to retarget the block time to 10 minutes, it's no more complicated than that.