Ah, you started with the retail terminal scenario; I started with the scenarios in use today thinking it could be expanded to those if the cost could get down low enough.
If I were to start with the terminal scenario I would have still do a smart card for form factor and cost reasons; implementation wise I would do a custom card applet that implements the bit coin wallet, communicated with a secured pin entry device (ped) or had onboard display and input mechanisms.
The approach you mention would work but I don't know if it could ever be scaled out to a currency card in a cost effective manner.
That said our two lines of thinking are compatible.
Actually, I started by thinking of ways to get my keys off of my home box, and ways to make sure they never ever had to be exposed to a hostile environment (disregarding loss of possession for now). Once I got going a little bit, I realized that it would work just as well at a retail POS as it would in my home.
You are probably right that it wouldn't be cost effective for the masses. I'm thinking around $100 in parts for the first crude ugly prototype, plus many hours of labor. I'm sure plenty of people here would pay that much, or double that, but we are not typical.