I've commented on this several times. It's the noise. My first prototype board didn't have the ferrite beads for both USB and PLL (clock) power supply. I scoped some serious noise on the GND plane when connected to my notebook, and when connected to the RasPi the noise was gone, and the error rates dropped to <0.7%, even at 330 MHz. How far the clock can be pushed when the new boards are made is still to be seen, but the error rates were the result of noise injected from the USB connection.