If miners leave for BCH, difficulty will increase which in turn will reduce difficulty for BTC (pulling miners back again). I don't think the majority of miners will flip/flop like that.