It's something to do with L3 cache on Intel CPUs, it doesn't scale well with more cores. I observed similar problems with 6-core vs 4-core Xeons.