The seed phrase isn't saved in the hardware device, but the master private keys are. So, if you don't use a strong passphrase, you can have your funds confiscated without your permission.
That's why I said I would assume it is safe. It is very possible that the hardware wallet can be compromised, it depends. But I can still just consider it to be reported, unlike paper wallet's private key.
Seed phrase are stored on hardware wallet, that is the function of secure element, or microcontroller unit in wallet like Trezor, just that it is hard to reach or compromised if stored on secure element.
That is true, using a strong passphrase on hardware wallet just extra protection would help against attack.