Your concern can be addressed by using the multisig wallet where HWs act as cosigners. The most secure setup involves the quorum of devices from different manufacturers, so that each one serves as a safeguard against potential compromise of the others.
True, but this doesn't help if every SE vendor is compromised and the kind of threat model I'm envisioning this is a credible scenario.