With a display, at least, there's no reason you couldn't add Bitcoins the same way you spend them. Even multi-sig backups could probably be made to work without a reader as well.
I don't really get it. If you don't have a smart card interface to your computer, how do you load any coins onto them? By going to some centralized bank? Certainly the grocery store isn't going to offer deposit services for bitcoin cards. If you
do have a card interface for your computer, you could set up your pin securely from there and load whatever coins you wanted from your online or offline wallets.
Personally I lean towards the contact smartcards rather than RF. They are more reliable, and less subject to tampering. It's less convenient, but not really any less convenient than credit cards currently.
I don't see how a contact based smart card would be any better than an RF card security wise. Neither has a large enough range to allow someone to steal your money remotely, and in both cases you're trusting that the manufacturer of the card reader has made it so that the merchant can't charge more than they said they would without retyping your pin.
AFAIK (from the few RF smartcard readers I've seen, and McDonald's is about the only place I've seen them) most RF card readers don't even require even so much as a pin input, which would be a disadvantage vs contact cards, but the card itself could require a pin to mitigate that problem. On the other hand, requiring a pin would be a disadvantage for small token purchases, like the original usage of smart cards as subway tickets.
Although the price tag of a smart card is way more affordable than a smart phone, I don't think they're well suited to the technology and usage of BTC.