Miners (and therefore the difficulty) react to the value of the money that they receive in the block. That's the block reward and the transaction fees, which obviously is going to add up to a lot when the price is very high.
It's how BTC becomes so secure - the more electricity that's "wasted" the more difficult it is to stage an attack on the network. Of course this means that it's arguably actually "wasted" energy, since it does serve a clear purpose.
More and more miners entered the market, but the environment always changed towards more difficulty, and this is what made miners change CPUs to GPUs as a main tool, and some time after that, create first ASICs.
The difficulty reacts to miners, not the other way around. And indirectly, the miners react to the market.