Thanks,  that's working much better I am getting around 1.4Mh/s with each of the 4 miners, and the hardware error rate has dropped right off, <1% so I can run the 328 clock again. Here I was thinking that the Pi wasn't powerful enough for USB 4 miners at once, but it obvious now that it is.

Not sure what the problem was with bfgminer, I tried 4.3.0 and 4.4.0 and both were terrible with multiple miners on the Raspberry Pi, fine on just a single miner. I will stick with this cgminer 4.3.5 fork for the time being, best performance I have had so far.