It turns out the controller is so low powered that it can only process ~30 shares per second which mean that unless diff is set higher, they work their way into a death spiral before diff catches up. Great...
(quoted from another thread)Is this a problem if one is using an S4 with solo.ckpool.org?
No. S4s work fine here.