With a display, at least, there's no reason you couldn't add Bitcoins the same way you spend them. Even multi-sig backups could probably be made to work without a reader as well.
1. You load simply by sending BTC to the address of the card - you don't need a smartcard programmer to load it.
2. The card receives a transaction request and if you see the right checksum returned from the card, you give your pin.
3. Provided the pin, the card now signs the earlier transaction with a unreadable private key on the card and sends this signed message to the terminal.
4. The terminal publishes the signed message to the network.
ANYONE see any holes in this?
shameless plug: https://en.bitcoin.it/wiki/Smart_card_wallet
Yeah seen it; already added my proposal with checksums instead of LCD screens (not that those aren't cool! I had no idea that could be done) 
Anyway we can create a complete BitCoin economy system here:
1. Bitcoin client is your bank and online payment device.
2. Smart card is your wallet and credit card.
3. Simple terminals accept payments.
That's it! And its physically impossible for a merchant to hack your card.
Lets do a mailing list or facebook group with people who support this and have programming skills.
I'm busy now, but in the summer I can program for this project.
Since ?Cascious? has done this sort of thing before he should probably be the lead dev and we just help where we can and with promotion.