... so he would recover the non-passphrase wallet, but by doing so, it does not mean the passphrase wallet is now inaccessible to the victim ( who still knows the passphrase + 24 words )
Take note that the two wallets (the one without passphrase and the one with passphrase) are two different wallets with completely different addresses.
Anything that happen to each of these wallets doesn't affect the other wallet.
If someone has access to the seed phrase and doesn't know the passphrase can only steal the fund in the wallet without seed phrase.