I've seen few people build their own Trezor and Blockstream Jade.
Aren't the components still black boxes? I've seen more discussion about potential bias in dice rolls than about potential problems with hardware wallets.
The fact that it's always recommended to buy directly from the manufacturer makes me think there's a lot of trust involved.
I don't engage much on electronic field. But on some cases, you can choose various electronic board/module, where i expect few of them are fully open source. For example, Blockstream recommend 8 different board/module if you want to build your own Jade[1]. At minimum, i do not expect backdoor designed to steal Bitcoin on such general electronic board/module.
[1]
https://github.com/Blockstream/Jade/tree/master/diy