That's it! I could read in the code that the minimum difficulty target is 0x00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff. That is transformed into 486604799 when compressed. Then that's why the difficulty never went lower, even when the blocks were produced slower than 2,016 blocks per 2 weeks.
Thanks!