I will preach for my church : yes, hardware wallet should be open source AND use hardware that is built on an open standard.
Just like the Satochip hardware wallet...

Maybe sometime in the future when we have a large developer community. For hardware wallets, the unfortunate truth is that releasing source code makes it easier for hackers to detect loopholes and carry out attacks.
Yeah, right. But in the other hand, if the bug is found by an ethical hacker, he may push a corrective patch instead of trying to corrupt some hardware wallet.
This is a good example :
https://satochip.io linked to the Github repo :
https://github.com/Toporin/SatochipApplet/releases/tag/v0.9-0.1