Will the person be able to withdraw your money or change password and block you from having access to your wallet?
Take note that your password encrypts your wallet file locally, so that someone who has access to your device can't access your fund.
Someone who has access to your seed phrase can use it to generate the same wallet and spend your fund. The thief doesn't need your password at all.