The difficulty will NOT go down as there is a "ice-age difficulty bomb" in place - "it's programmed to raise difficulty exponentially".
Difficulty does not only depend on the hashing power but also on the number of blocks mined and times passed since the last update.
Difficulty bomb graph: https://etherchain.org/charts/difficultyBomb
Difficulty calculation: https://gist.github.com/CJentzsch/c78768f9837afb8eef74