If an intruder wishes to attack a wallet it is expected of such to carryout 2128 operations to guess a 12 word phrase.
It is far more difficult than that and not feasible for a hacker to correctly guess the entire seed phrase of an address, even without an extra passphrase as it's more complex than simply guessing any random 12 words.
Also, a seed phrase can be used for plausible deniability; to create an address which is different from the original one, this can be used to hide balances should the holder find themselves in any trouble.