Which is why some schemes allow you to add password to your seed, so the seed alone is not enough to access the wallet, because private keys are derived from both the seed and the password. I personally don't use this method, because I use Electrum and it doesn't have this option, but even if I could, I still wouldn't because it just adds just the complexity of storing password somewhere. I'd rather not risk locking myself out of my coins because of a misplaced password.
I think Electrum wallet also has the feature of Passphrase. Instead of storing your password anywhere, you can choose a strong password which you can remember. Even if the seed is stolen, without the passphrase they cannot access the wallet.
You can choose to keep part of your funds under passphrase(large amount) and fewer amount without a passphrase(small amount). If the wallet is stolen, then only the small amount without a passphrase will be lost. You can still save the large amount of funds.