I was talking more about security from merchant overcharging, and for that matter, since BTC is spent by referring to previous txOuts, you'd have to load all the txOuts onto your card, or trust the merchant to construct a tx with the proper amounts that pays you the proper change. If the merchant is using a USB smart-card reader, they would have even more room for messing with clients since there are no software restrictions on a general computer as opposed to a specialized card payment unit.
That's exactly the point. You can put multiple addresses on a smart card, verify the transaction amounts with a display, and create all the transactions right on the card. There's absolutely zero need to trust merchants.