With FSS, if the rest of the network sees my malicious double spend first (which is how the attack takes place), then they will refuse to replace it even with the higher fee paying multi-party transaction (coinjoin, Lightning channel, etc.)
I don't understand this part. Since you've let a non-RBF transaction been propagated, how can you cancel it with an RBF unconfirmed parent?
hosseinimr93 is correct. For further expansion:
I make Transaction A, which spends UTXO 1 and creates UTXO 2, and is opted-in to RBF. I then use UTXO 2 to join a multi-party transaction. At the same time, I make Transaction B, which spends UTXO 2 and is opted-out of RBF, and broadcast Transaction 2 to the rest of the network. As the rest of the network has seen Transaction 2 first, which is non-RBF, they consider the multi-party transaction to be a double spend and reject it. Once the affected party has maxed out their fee reserves with repeated CPFP attempts, I can then replace Transaction A (which is RBF enabled), which means UTXO 2 no longer exists, means Transaction 2 is now invalid, and the rest of the network will now accept the multi-party transaction and its excessively high fee paying CPFP children.
I might be missing something too, but I don't think the purpose is merely to annoy/harass the other parties, but to delay confirmations until the RBF/CPFP algorithm gets close to maximum fee range, then the attacker sends his own RBF transaction and forces everyone to pay that high fee.
It can be either - wasting time or wasting fees.
The attack vector would require a a miner with a lot of hashing power to take advantage of the attack.
It doesn't require any mining power at all. The attack can take place as I've described just above in this post.