...
Of course it has the problem of abuse by larger miners splitting up their 'farm' to multiple addresses and thus acting like multiple smaller miners - so that would have to be resolved some way I've got no idea how

It wouldn't be in a large miner's financial interest to push out/hurt smaller miners. Just makes p2pool network speed lower for everyone. Granted, the community is full of vandals though.
Edit: Correction, the hard code in p2pool was to make a miner's minimum difficulty target even if they said /1 to be such that they can only find 1.67% of the total share chain's shares. This is far too many shares for 1 miner to have, of course...
So basically what you are saying is that people with more mining power will be happy to have MUCH higher difficulty than p2pool's already very high difficulty, so that others can have lower difficulty, and thus people with more mining power will also have higher variance than most people mining on p2pool so the others can all have lower variance?
Unfortunately "people will do what is best for others at their own detriment" usually doesn't pan out too well.
In fact there's a very obvious example with the recent blocks where 2 were mined by people who put almost no transactions in them.
No transactions confirmed is bad for bitcoin in general, but anyone short sighted may well only care about the BTC reward and not the bitcoin network ... or p2pool's reputation.