next difficulty is based on total last hash power solving the last block, am I right?
No. Its the average time interval between each of the generation of the last 2016 block at the point of difficulty adjustment.
is it possible to make the difficulty adjustment more responsive and adapt faster?
Yes. You have to switch to a different algorithm like Kimoto Gravity Well which does have quite a few downsides. It isn't especially useful for Bitcoin and it should not be implemented nor anything regarding the difficulty should be changed.
by the recent event, I can see the difficulty stayed high even when lots miners abandoned BTC
that resulted a big gap between blocks because the imbalance of hash power and current difficulty
is there a way to improve difficulty retarget so that miners hopping would be less profitable
Uh. It was more about the rise in the price of BCH. If it were to keep rising, miners wouldn't consider mining Bitcoins at all.