If miners leave for BCH, difficulty will increase which in turn will reduce difficulty for BTC (pulling miners back again). I don't think the majority of miners will flip/flop like that.
it is not just the difficulty. right now mining bitcoin is ridiculously profitable. they are earning about 13-14.5 bitcoin per block and that is worth $54,000-$60,000 per block. and price will stay up and the swings will be 10-15% at worst case scenario.
meanwhile in bitcoin cash they earn 12.5-12.8 BCC and that is $9,000-$10,000 and that is while the swings are crazy as big as 60-70% loss in one day.
a miner getting the reward for bitcoin can be sure he will get the $60K when the 100 block period passes. but the miner mining BCC can not be sure he gets $10K when the 100 block passes. by that time price may be back at $200 again.