Difficulty wouldn't decrease, it wouldn't change it all. Difficulty adjustments only occur every 2016 blocks and those blocks don't mine themselves. If there was a small amount of hashing power still online, then it would take a long time for a difficulty adjustment to occur. What you want are small drops in hashing power.