Is the passphrase on the wallet or is it actually on the seed mnemonic.

If it's the mnemonic then it should be temporarily fine until you move the funds to a new wallet. If its the passphrase it asks you for before sending funds that you haven't released then the other person might have access to your funds if they have a way to access the seed - if they just saw it then it's unlikely they memorised it.