For higher hashrate miners, is it suggested to manually set the difficulty in cgminer (with p2pool), or does it not matter?

For example, right now my cgminer instance is running at around 42GH/s. It has set the difficulty to 8. Is there any reason to set it higher than that? Reduced bandwidth/latency? My miners are on the same LAN as my p2pool but a different box.