The noise in the time it takes to generate blocks is very, very high.  There was a link posted somewhere in the forums that showed the time taken for the swarm to solve the previous 100 blocks (or so), and the range in solution time was between 3 seconds and 20 minutes.

Adjusting SLIGHTLY after each block makes sense to me though.  Maybe the difficulty can go up or down a fraction of one percent each block, and then recomputed wholesale on the two-week boundaries?

Regardless of how it's done though, we need to be wary of wild swings...