Anyone has the seed phrase needs the passphrase too.
Without the passphrase, your seed phrase will generate completely different addresses.
But now somebody can brute-force the extended words of the seed phrase. True, the existing bitcoin wallet crackers such as hashcat and btcrecover, do not support this kind of recovery with seed phrase input at this time, but a) the seed phrase can always be sold on the darknet, and b) it could end up with someone who has a custom tool for solving this kind of stuff.
The last thing you want to happen to you is having your seed phrase end up on Google Search. I recommend moving all your funds out immediately before they get stolen.