Exactly why does the small difference in price matter so much? Are you instantly selling your coins? Then this wouldn't come as a surprise to me.
Why not keep mining all the time and sell it later if you have to?
I remember the days that I got into this business, the difficulty was 7M (!) or even less. It was quite some time ago.
If I only had a 4 TH/s ASIC then.