git branch says:
* libusbx
The error messages happen every 2 seconds. Using top or xload only shows every 1 second so I don't know how to see spikes.

CPU utilization is very low - something like 0.3% for cgminer.
Try the latest
libusbx async branch. More sheer desperation in the code. Then show us the output of api stats if you know how to (easiest is to use 'java API stats') after it's been running for a while.