How did you lose money? Did you pay him while your incoming transaction had 0 confirmations?
The part where his first transaction didn't confirm should have been a dead giveaway: it was double spent.
That is exactly what happened, because I delayed the transactions for multiple hours I decided to do it fee free and as fast as I could when I had access to my wallets, that is why I did not wait for it to be confirmed. I got good news tho, please read the section at the bottom of the original post.
The scammer probably abused the RBF feature.
If a transaction has been flagged as RBF, (as long as it is unconfirmed) it can be replaced by a new transaction spending a same input and having a higher fee rate and higher absolute fee.
In your case, the fee rate used for the original transaction was 1 sat/vbyte and it was replaced with a new transaction using the fee rate of 3 sat/vbyte.
In the replacement transaction, the scammer sent the fund to his/her own address.
Thank you. After doing some more investigating this seems to be the case, I wasn't sure at first but your reply confirmed it for me.