Will someone have access to your wallet without your device but the person manage to see your private keys and copy them without your knowledge. Will the person be able to withdraw your money or change password and block you from having access to your wallet?
When someone have the seed phrase of the wallet that you are using where you are storing your funds then Yes, someone will be able to steal your money. Seed phrase is enough to access the wallet and the best choice you do is create a new one and transfer the funds right away if you think someone knows the seed phrase.
It's important that no one knows about it and how you keep a back up of the seed phrase and where you save it.