Any soft fork should be backwards compatible, unless there is a good reason not to. ... Even for security and bug fixes, the objective should be to not make any transactions invalid.
That is mathematically impossible. A soft fork, by definition, is a change that
only makes the rules more restrictive: that is, some transactions that were valid by the old rules are invalid by the new ones, whereas all transactions that are valid by the new rules are also valid by the old ones.
Barring emergency fixes, you can make it so that the change depends on the transaction version number.
As I explained already, that is often not an option. Soft forks are often issued precisely because it is necessary or desirable to outlaw certain types of transactions. Note that miners cannot distinguish a genuine mothballed transaction from a new transaction that is using the old version number just to frustrate the fork.