on the other hand I know someone who can do something and get my eth back(eth which I bought these tokens with),for that matter he need the control (with my wallet secret phrase).
They don't need your passphrase to simply send coins. If they need to send a custom transaction from your wallet, it still can be signed on your own computer. If they want to confirm that address of yours, use
arbitrary message signing. Don't ETH wallets support all of this?