Will someone have access to your wallet without your device but the person manage to see your private keys and copy them without your knowledge. Will the person be able to withdraw your money or change password and block you from having access to your wallet?
Can.
Initial phrase is their first way to spend your assets because the initial prhades are different from passwords.
Because the initial seed and private key are very important, the OP needs to guard it from the reach of others.
Maintaining early phrase and private key safely is the same as keeping ourselves survived the danger.