Matt, we're referring to p2pool peers, not bitcoin peers. I think that the p2pool blocks (the "shares") are very very small in comparison to bitcoin blocks. Since my node is the origin of the newly found share (the share is outbound to the rest of the p2pool network) it can only get out as fast as the maximum outbound bandwidth of my node - it seems to me that I want to try to hit that max (which in my case is 250 Mbps) so that there are many spokes to my hub (this seems to me to be hub/spoke topology in the first set of share transmissions, then it turns into a graph as the share gets retransmitted by my peer nodes).
Am I totally off base here? I'm not an expert on p2pool, I just read some stuff.
Indeed, the relay network is a bit different. As many of its peers still use standard bitcoin p2p connections, full blocks get sent over the network, making peaks much higher. Still, you have to think about more than just your node. If you found the share, maxing your uplink isnt a bad idea, but if someone else found the share, you still want their share to propagate quickly (at least to you...), and for that you shouldn't fill their connection count. Of course you also have to account for bitcoind being on the same uplink (what if your share was a block?).