Is there something we are over looking here? With a much higher (actual) hash load will this adjustment act differently? What are the dangers of allowing very large diff swings like this?

Also, with a higher difficulty # like the 10-20 range we currently see, will we see any other phenomena that we can't test in the low hash test-net environment? Are we on the right path? What are your thoughts?