Can the difficulty go down as well?
I tink it depends of the coins. For bitcoin, it can not. even if you stop mining.
That is not correct. Difficulty can (and has) gone down.
If the hashrate of the network doubles, then the time between blocks will be cut in half and next adjustment difficulty will double.
If the hashrate of the network is cut in half, then the time between blocks will double, and next adjustment difficulty will be cut in half.