Mathematically, assuming an average block time of ~10 minutes, there are a maximum of ~104 difficulty adjustments over a 4 year period, so even if there was a .01 MB increase at every difficulty re-target (the chances of which are negligible), the base blocksize would still only be ~2.04 MB after 4 years.
Is this a compromise most of us could get behind?
For me, maybe
But bear in mind, saying 2.04 MB after 4 years conceals the fact that the real total blocksize would be 8.16MB when you include the signatures in the witness blocks, Segwit is a part of this deal you're proposing.