Some Hardware Wallet manufacturers are using Microchip ATECC608A are they claim Firmware they claim it is open source. Passport wallet  released everything on their github.
Same chip is used for other devices, not just hardware wallets, and most of it is released on github, but full datasheet is under NDA:
https://github.com/MicrochipTech/cryptoauthlib

No chip or  secure element is perfect and there will always be some bugs, but I am not so sure about government backdoors.