This should not matter. When pool rolls out new diff, this diff is used with new job - official stratum mining specifications say that:
This means that difficulty 2 will be applied to every next job received from the server.
If the pool does not respect this stratum rule, you may get lower hashpower.
Slush mentioned it here too:
https://bitcointalk.org/index.php?topic=557866.msg6078264#msg6078264