I will preach for my church : yes, hardware wallet should be open source AND use hardware that is built on an open standard.
Just like the Satochip hardware wallet...
Maybe sometime in the future when we have a large developer community. For hardware wallets, the unfortunate truth is that releasing source code makes it easier for hackers to detect loopholes and carry out attacks.