The sweet spot is probably 16 and 32-bit microprocessors: fast enough to handle crypto without pain, small enough that hiding malicious features is very tough for the manufacturer, and cheap enough that the community has a chance of auditing the actual shipped hardware and firmware against the claimed design.
It shouldn't be a microprocessor at all.
Think something like this, but to perform ECDSA instead:
https://en.wikipedia.org/wiki/555_timer_ICTruly secure hardware wallets will need to be built at the level of electrical engineering, not software engineering.
I am an electrical engineer, and frankly what you're saying is crazy.