What server are you talking about? Are you talking about Spot client or are you talking about some pool?
Difficulty adjusts only based on the time it took to mine last 2 blocks and not based on anything else.
the network server. If SPOT uses only 2 blocks to calculate the difficulty that would explain these jumps.