The new algorithm take care of this, this is why it has been implemented. If big miners left, the difficulty will auto adapt very quickly, so "small" miners doesn't have to "pay" for this. Inversely if big miners comes, diff rise very quickly too symmetrically. No more "stolen" coin is allowed now, so big miners don't have to "come and leave" anymore.