But I can understand the whole point of RBF and that is to allow double spending [only one confirmed transaction] and allow us to pay more fees to get fast confirmation for the same transaction.
Not sure if that's a typo or you've misunderstood, but RBF is only possible on unconfirmed transactions, not transactions with one confirmations. Also note that although your replacement transaction might end up paying the same addresses the same amount, just with a higher fee, the new transaction is very much a brand new transaction with its own transaction hash, and is not "the same transaction".
However, regarding the issue of double spending or redirect a transactions to other wallets (own wallets or other people's wallets) it will certainly be a threat to anyone who makes buying and selling transactions without involving escrow
If you wish to accept 0 confirmation transactions, then you need to ensure RBF is disabled. If you receive an RBF enabled transaction, you should wait for a minimum of 1 confirmation.