This is why I always send my transaction with a custom tx fee that is adjusted manually. I lowered the fee to my desired amount and just enabled RBF just in case the situation needs me to confirm the transaction asap.
If you are sending funds to your own address or consolidating your inputs, then you can simply attach a low tx fee and wait for as long as possible for it to be confirmed, as long as you do not need the funds. But people who are making payments or sending the funds to another party cannot do that, because you want the tx to be confirmed as soon as possible, and in that case you have to pay more in fees, which is sadly very high right now.