Your wallet is merely a tool to interact with coins stored inside the Bitcoin blockchain, and this tool is almost useless without the actual keys that you mentioned as a "seed phrase." Anyone who has gotten access to your secret key can theoretically import it into another wallet and move your coins to an address to which you have no keys. This is called a transfer of ownership, and it actually occurs every time you are making a transaction with another person. The main point of this story is that coins are never stored locally on your device: instead, they are being held by thousands of computers globally, but only the person who knows the secret can move them around. This is why it is important to never disclose the location in which a backup of keys resides or share the key itself because it would mean that you are no longer the sole owner of coins.