Correct me if im wrong...
Worker difficulty settings its all the same at the end. Just pool determines how many shares have to give u depend on ur khash.
Pool dont like to communicate 50 times per second with u for example just a few times so it gives u a share that will make that time happen.

My english sucks i hope u got it ...