Right after the 2020 halving, there was a dip in Bitcoin mining difficulty. The network lost some trillion hashes in just a month's time. It was probably due to the halved block reward. Some miners might have been compelled to pause operations due to the decreased reward. But then it quickly recovered thanks primarily to the fast increase of Bitcoin's price. It only took around 6 months from the halving day for the price to double. So it balanced the halved reward. Miners must already be in profit. From then on, the difficulty climbed more or less continuously except when China announced a ban on Bitcoin mining.

So, there could also be a temporary dip in difficulty after the next halving takes place. But recovery will quickly follow.

As to the possible backlash in general, I don't think there will be any. This has been the design ever since. But regardless of a possible event where some miners stop operating because of losses, the difficulty would just adjust and the network continues to operate as always. And a significantly lower difficulty will eventually be enticing for those who want to mine. The price of Bitcoin will also play a vital role.