You are right but not totally true. Like if using hardware wallet like Ledger Nano, it has a secure element but which is close source, the secure element makes it impossible for hackers that steal Ledger Nano wallet to get access to the seed phrase, if the seed phrase can no be accessed, then no access to the private key. So the thieves will only steal the device for nothing.

It is very possible hackers can steal Trezor but able to access the seed phrase, this vulnerability has been discovered by Krakn like two or three years ago. But the use of passphrase to extent the seed will help for such hackers not to be able to access your cryptocurrencies  stored on Trezor because salting process in which additional words (passphrase) are used lead to generation of different keys entirely which will make it impossible for hakerd to get through to steal your funds.

There are some uncommon hardware wallets, example is the Coolwallet S that do not support passphrase and yet such physical natyqxk can lead to access to its passphrase, this will only make hackers to steal such wallet to get through by knowing the wallet seed phrase and has access to the keys generated by the wallet. That is why it is good not to use uncommon hardware wallet.