Correct. And criminals will just shift their focus to something else.
Back in the days, bank robberies or even train robberies were a thing. Nowadays, banks don't have cash, and barely get robbed. At least in my country, banks don't have bullet proof glass anymore, there's just nothing to get. Robbers shifted to gas stations, and now their employees have to hide behind bullet proof glass.
If cash is banned completely, they'll find something else to steal. Here, trucks are getting robbed now, usually when they're parked, but also while driving. Small electronic items are high value targets.
Or they can rob people at home: "your Bitcoin or your life".

I'd be fine with replacing cash by Bitcoin, but the current shift is from cash to electronic money. The main problem is that at least printing cash has some limitations, while (central) banks can create virtually infinite amounts of electronic money, robbing people through inflation instead of at gun point.