Wouldnt it be better if Bitcointalk provided a common test environment that was open for anybody to use, without people having to download the software and setting up their own environments? Depending on how the software has been developed, it could be on "debug mode" and trap software errors into a log in its own. Functional errors would logically need to be reported by the people who test the software.
Testing the software has multiple perspectives (installation, functional usage under different profiles such as admin, mod, regular user, db admin, etc.). Surely here the main target would be that of regular users (functionality and stress tests), being the other profiles more delicate and subject to perhaps a delimited cherry-picked set of volunteers. But a common forum created test platform would seem the way to go.
There was a test forum that was disabled
http://beta.bitcointalk.org/I think that theymos proposal is not just that we test, is that people try to build another forum using the same software. And try to correct and modify what each one finds interesting. It seems a more practical idea than centralizing the modifications and tests in a single version. He does not have so much time to test everything that was proposed. So he could just modify what was already working well and the transition would be faster.