assuming you can even implement a protocol that doesn't allow the private keys to be leaked
A lot of smartcard apps are poorly designed. But it isn't black magic or anything. It's definitely doable. Look at the satellite TV access cards. They can be reverse engineered, if you have access to the card itself and a scanning electron microscope.
if you can install an overlay between the keys and the actual circuit board, you can easily capture the pin, and launch a replay attack.
a much better way is to have a portable wallet that "pays" a merchant by transferring a signed tx, which the merchant can verify and broadcast.