Sometimes the diff gets adjusted to values less than the pool minimum
That's not possible. It's a minimum value (42) because, by definition, that's as low as the pool will go.
What specific values are your machines adjusting down to? If your shares are getting accepted then there's no issue here I can see.