This is the answer.
on p2pool, load average: 0.98 0.92 0.87
vs
ozco, load average: 0.19 0.16 0.20
You do understand what the load average figures are telling you, right? It is simply a count of how many processes running on the system needed CPU time per second during that interval, it does not tell you anything at all about the actual CPU time spent servicing those processes.
Eg, if I do a
while /bin/true; do echo "."; sleep 1; done
The load average will show as 2,
No it won't. The load measures the number of processes using CPU, waiting for CPU or waiting for I/O system calls to complete (basically reads and writes on file descriptors).
even though you can probably see that the actual amount of CPU time spent was likely < 0.001% ?
A high load average may be an indication of the system being CPU starved, but I'd judge "high load average" to be 10-12x the number of CPUs present in the system, depending on the type of processes the machine is running.
Load > nb of CPU threads may not have anything to do with CPU, just fork large number of processes writing to disk, they won't use much CPU on modern hardware (where every I/O use DMA) but your load will be as high as the number of processes you forked.
In the case of cgminer on Avalon with p2pool, ckolivas already replied in an earlier conversation that p2pool makes cgminer use more CPU than traditional pools. This doesn't show on "normal" CPUs but Avalon has a very weak one.