You should look at smartphones.
Already in Japan and Korea they have skipped the smartcard and people are now paying by waving their smartphones at the counter (no idea how it works).
Also, have a separate wallet for this concept as a whole then security is not really a concern.
It's like my current smartcard used for public transport and sometimes other services that can be used in London (Oyster), Hong Kong (Octopus) and many other cities. You don't store your entire bank holdings on it, just enough for a week, say $100 max.
If you encrpyt it good enough, no hacker will bother to spend time to just get 10 bucks on the card.