Thanks for the responses.
I switched to spacepools and have been doing much better.

I was on Nanopool with that problem, on the highest difficulty port. I suppose that setting a static difficulty would address the issue there then.