If nothing else the huge pools could agree to stop mining once we hit 2000 blocks and let the smaller pools find them, and when the difficulty changes, won't that have an affect on what the new difficulty is?

That is about the same as asking Walmart to shutdown to allow the local Mom & Pop stores some business.