Which is why some schemes allow you to add password to your seed, so the seed alone is not enough to access the wallet, because private keys are derived from both the seed and the password. I personally don't use this method, because I use Electrum and it doesn't have this option, but even if I could, I still wouldn't because it just adds just the complexity of storing password somewhere. I'd rather not risk locking myself out of my coins because of a misplaced password.
I think Electrum wallet also has the feature of Passphrase. Instead of storing your password anywhere, you can choose a strong password which you can remember. Even if the seed is stolen, without the passphrase they cannot access the wallet.
You can choose to keep part of your funds under passphrase(large amount) and fewer amount without a passphrase(small amount). If the wallet is stolen, then only the small amount without a passphrase will be lost. You can still save the large amount of funds.
Usually people don't own only one single wallet, they possess several wallets with several seeds and even several lonely private keys belonging to one address they've used to receive a payment or signing a message. So what is your advice when you have several seeds? Do we need to use one single passphrase for all of our wallets or do we need to change it and use a different one for each of our seed/wallet?