On Phillip's idea to assess miners by the hardware they use:

As a HW consumer and pool user, I would love that comparison, but I don't see how that can be done in practice in anonymous pools.  Moreover, the performance capability of miner is affected, as Minor_Miner points out, to hardware and software, as well as clocking conditions.

I just don't see a way of validating that users are grouping their hardware correctly.

The simplest / cleanest way of validating a user's hardware is to be able to periodically and randomly test a user's hardware / hash volume with a test case (like drug testing for athletes). A larger user should have to demo hashes proportional to their size.

The next level (but more complex) option is to group users into larger buckets (5 Th sub groups and 250 Th groups) to assure one bucket is behaving in statistically consistent ways, with randomization of the subgroups periodically to ferret out weaker players.  This idea will go to address the viability of the hardware as we climb the difficulty curve.  It is conceivable that hardware that works at lower difficulties breaks at upper difficulties simply by the way the HW/SW combo hashes.  We are looking for unexpected / forseen issues with hardware.

I expect the latter problem to go away as the payouts diminish with time, and the power hungry machines start to lose money on an operational (electricity cost) basis.  The more mature stuff *should* be better tested.