It is very possible hackers can steal Trezor and able to access the seed phrase, this vulnerability has been discovered by Krakn like two or three years ago. But the use of passphrase to extent the seed will help for such hackers not to be able to access the cryptocurrencies because salting process in which additional words (passphrase) are used lead to generation of different keys entirely which will make it impossible for hakerd to get through to steal your funds.
There are some uncommon hardware wallets, example is the Coolwallet S that do not support passphrase and yet such physical attack can lead to access to its passphrase, this will only make hackers to steal such wallet to get through by knowing the wallet seed phrase and have access to the keys generated by the wallet. That is why it is good not to use uncommon hardware wallet.
Secure element is designed to never leak the seeds or to at least make it inherently difficult and/or expensive to access it. Passphrase is used as an additional security measure against attackers if that layer of defense is broken, plausible deniability as well but using a passphrase is not desirable in all situations; not being covered by checksum, forgetting it, etc. AFAIK, CoolWallet has a secure element which makes it that much harder to extract the seeds in the first place.