Your concern can be addressed by using the multisig wallet where HWs act as cosigners. The most secure setup involves the quorum of devices from different manufacturers, so that each one serves as a safeguard against potential compromise of the others. For even greater resilience, it's a good idea to have in quorum those HW that use electronic components ^{such as SE, MCU, displays, etc.}from different vendors and employ different programming languages in their codes.