Strangely enough, this is by design and it's working as intended.  Smoother changes in n-factor can't really be coordinated without opening up new attack surfaces.

The issue is that it takes a long time to hash high n-factors and a short amount of time to hash low n-factors.  The only real parameter that can be tweaked there is the difficulty, which helps to balance things -- but it can't be too drastic either.  In most coins, the amount the difficulty can change (in either direction) is tapered so that the swings are not so large.  Apt-curve specifically addresses that for  this coin and allows significantly larger swings as calculated to keep the average amount of time for the last N number of blocks to match the target time.  But that's an average time across N blocks, not the time on average for any particular n-factor.  It still means higher ones take longer and shorter ones are quick.

If you think of it as transaction processing, it's true that the network can appear stuck for some time while working on a high n-factor, but as transactions pile up, they are all cleared shortly after unless the network is really unlucky and gets a string of high n-factors in a row.

The n-factors are truly random and are computed based on the winning hash of the previous round.  It's crucial that it can't be predicted what n-factors are upcoming otherwise miners can easily attack the network by jumping in on the scheduled low n-factors and jumping out for the high ones, leaving the chain completely unviable at a high difficulty.