There is another situation where the difficulty could go down. You se it all the time in altcoins like litecoin. If another SHA256 coin we to be more profitable to mine people would point their miner to that coin reducing BTC difficulty. But I don't se any SHA256 coins even remotely close to BTC. Most alts are script to try and avoid the arms race we se in BTC now. lots of competition in scripts no competition in SHA256.