The difficulty never changes by more than a factor of 4 (except for in a special case that only applies on testnet).

The code that enforces this is

https://github.com/bitcoin/bitcoin/blob/d62a1947be5350ed60066ccacc7aba43bbdf48fb/src/main.cpp#L875.