That's exactly the point. You can put multiple addresses on a smart card, verify the transaction amounts with a display, and create all the transactions right on the card. There's absolutely zero need to trust merchants.
How do you get them on there? In order for a card to create tx it would have to have the relevant txIns, in which case it could also know its own balance, and could use a simple interface for "locking in" the settling price for a tx prior to pin input. Again, though, that requires having a computer and a card reader for loading and managing it, or else a bank which can do it for you.
Hmm, I see your point. If the card is your only device, you do have to trust whomever you purchase Bitcoins from. That's not entirely different from most Bitcoin users today, though, who have to trust the exchanges they send money to.
But perhaps it could be solved by setting up some sort of centralized service that would just send you a signed verification of your balance. That way, the card itself could query the service through a 3rd party terminal, and would only have to trust the service, but not the terminal. The card would have the public key for the service, and can verify that the balance was not tampered with.