Sorry for potentially misunderstanding the problem, but why is hardware - in any way - dictating the direction of software development here? Bitcoin itself is built to scale with changing hardware capabilities by automatically adjusting difficulty as needed. What makes a decentralized mining pool so different conceptually?
Well, I'm not lying to you when I say that the p2pool developer changed the system to make it possible to use the type of hardware design that the first ASICs used. People tried using AsicMiner blades and Avalons on p2pool when they first came out in Spring 2013, and they didn't work. Forrestv changed the share interval and it's overall period to allow for it. So there's one way that hardware has dictated the direction of p2pool's design. Kind of pretty significant really, there was a lengthy hand-off period between the old scheme and the new scheme, after which the nodes running the old scheme were cut off from everyone running the 30sec/72 hour version.
The current range of hardware can only perform with the characteristics that it has been designed with, and I'm not sure to what extent that was to simplify the development, the design, or to help conform to standards in the protocol (the mining protocol changed shortly before ASICs, to enable their efficient usage of network bandwidth, and I believe we have two standards to replace the old CPU and GPU proficient standard)
What the challenges are with changing the hardware to send and return solution attempts in shorter batches, I'm not entirely clear on. So I'm putting it out there.
Why do you worry about forcing out a subset of current miners who will be forced out with time (and increasing difficulty) anyway? If all miners were forced onto p2pool tomorrow and people with less than (as an example) 1TH/s rigs are no longer competitive, would the remainder of the hashing power distribution really be any less diverse in terms of control than the current situation with GHash.IO? It seems to me that with a bit of time, everyone currently mining is going to be "forced out" anyway due to the competitive forces of hardware advancement. If a software change temporarily accelerates that process in an unbiased way for everyone on the network, how is that bad? The only way I can see it mattering significantly is if the % of miners forced out is very high (e.g., a majority of all miners) causing a slightly "unfair" advantage for those with access to the best hardware (since the marginal cost for additional ASIC hardware could buy a slightly bigger piece of the pie than it could previously), but that was true when the first ASIC rigs became available anyway. The market (in my opinion) has and will adjust if there is any such economic advantage for certain classes of mining hardware.
When you think about the health of the mining network overall, you've got to consider the influence the players have. The big centralised pools get consulted to secure their cooperation with any issues on the network that require emergency measures (blockchain rollbacks, emergency patches being the issues that come to mind). Forcing small players out of the network to "decentralise" may only end up further consolidating the influence of the big players. In the growing climate for big private mining firms, both now and in the near future, protecting the share of the small players is important. What you're suggesting decreases the incentive to mining hardware manufacturers for attempting to produce anything but the most dense and voluminous concentration of hashing power in a unit. What if all mining devices ended up requiring 240 volt electricity supplies, and thousands of watts of heat exhaustion? So much for
vires in numeris.
We should do all we can to diversify the mining network to keep everybody pulling in the same direction, as the mining network is such a fundamental part of bitcoin's security and integrity. A small number of large firms with self run mega-mines puts too much decision making power in too few hands. There are many potential scenarios for that kind of situation to cause setbacks and imbalances. We should be aiming to make the network with a low barrier to entry, so that even very small players can have a stake in the gains and the rules that create the supply of this money. Promoting an environment that allows large firms to corner the market could put us all in a position not unlike the way money has been issued in the past. I became involved with cryptocurrency in part because it represented an opportunity to change that situation for the better.