Passphrase are not stored on hardware wallets that I know, if the robber was able to see the seed phrase, he can not be able to have access to the passphrase and not able to steal the coin.