A short period where the buffer is negative will likely not be detrimental. However, if the buffer at some point meanders quite negative, rational miners will no doubt begin to trickle away, as at that point, the chance of receiving full pay for your shares anytime in the near future is rather small. I expect this wouldn't become significant until the buffer went below -1000. But that's just a guessing game.