Tldr just move the funds to a new wallet as soon as possible and you feel up for it - not too tired to make a bad mistake.

Is the passphrase on the wallet or is it actually on the seed mnemonic.

If it's the mnemonic then it should be temporarily fine until you move the funds to a new wallet. If its the passphrase it asks you for before sending funds that you haven't released then the other person might have access to your funds if they have a way to access the seed - if they just saw it then it's unlikely they memorised it.

Anyone has the seed phrase needs the passphrase too.
Without the passphrase, your seed phrase will generate completely different addresses.

Although it's not possible to access your fund without the passphrase, I recommend you to create a new wallet and move all your fund to it as soon as possible.
I don't know how complicated your passphrase is. But there's a probability that it can be brute-forced.  

Edit:
As mentioned by jackg in the below post, if by passphrase you mean the password used for encrypting your wallet file or the password asked when sending bitcoin, then the seed phrase is enough for spending your fund.
What I said above is true only if by passphrase you mean the custom words or characters you add to extend your seed phrase.