Actually, the factor of 4 limit means that the attacker can initially mine the crap out of coins unless there is a hard fork intervention.  But the factor of 4 takes a pretty quick toll on their advantage.

Given the loose parameters laid out previously in this thread, let's say the attacker starts in with hash power that is orders of magnitude above the rest of the network.  How long does it take the rest of the community to recognize the attack, alert developers, developers formulate plan, developers implement plan and fork code, and sufficient number of miners implement the hard fork? 2 days maybe?  So if the attackers equipment is powerful enough to mine one block in say 10 seconds at the current difficulty, then it will take 20160 seconds, or 5.6 hours to mine 2016 blocks.  After that the difficulty goes up by a factor of 4.  I haven't looked at the underlying math, so what follows is an assumption.  The assumption is that the same attacker would then need 22.4 hours to mine the next 2016 blocks.  Total time so far is 28 hours.  After that the difficulty goes up again by a factor of 4 so (using same assumption) it should take 89.6 hours for the attacker to mine the next 2016 blocks.  The hard fork would go into effect about 1/3 of the way through that.  So the attacker would mine about 5000 blocks in the 2 days required to implement a hard fork.

If we instead say the attacker can mine a block in 1 second, then the scenario does not get much worse.  The times for mining 2016 blocks for the attacker go in steps of 33.6 minutes (2016 seconds), 134.4 minutes, 8.96 hours, 35.84 hours, and 143.36 hours.  In this scenario the attacker would mine just under 8100 blocks before the hard fork could be implemented.