Not that I ever did that, but I think that when you recover a wallet by a BIP-39 seed you need no passphrase, at least with some wallet software. Maybe it's worth checking if this is the case.
Using a Trezor which will need both the seed and passphrase to gain access the hidden wallet.
I meant that if it's BIP-39 and a bad guy gets the seed, you might be in trouble even if they don't know the passphrase. Maybe trezor doesn't use BIP-39 seeds, though?