The thing is that someone could possibly modify the hardware/software within the hardware wallet in a way that would allow whoever sold you the hardware wallet to know in advance which seed it will generate, or otherwise leak the private keys.
The software must be easy to check. But the hardware attack is an interesting problem.
Do you know any case of an hardware attack?
I think all hardware wallets are much too new to trust with large amounts of money. They have not been thoroughly tested against various attacks.
I guess most of the time these things are meant to be unplugged so the risk of an attack should be very low.