Why not implement maximum block size alongside mining difficulty adjustment using the same mechanism?

Rather than an arbitrary 20MB limit or rolling quadruple/exponential maximum size increases, why not incorporate a self-adjusting maximum block sized based on the number of the last n blocks solved that hit the existing hard limit?