the solution can be quite easy (in theory), just set high limit on hashpower from single miner (or single wallet recieve rewards), let it be 20% of total poll hash rate
That would be pool-side and can't solve the problem. It could only contain it. I suggested that pools do not provide dedicated ports for the nicehash botnet. Masari.superpools provides even two dedicated nicehash ports.
But even if it didn't, someone could always create a private pool for nicehash and it happened already. The solution must be network-wide on the protocol level.