Confirmation time can only really be an issue when you spend money on the high street - online retailers are happy to wait 1 hour before they ship something to you. Even on the high street I can't see it being a problem either, consider examples like this:

If you buy a takeaway coffee for 10mBTC, chances are that most cafes will be happy to let you leave with a 0-confirmation sale - the probability of fraud is very low, and the impact of the fraud is only the price of a cup of coffee.
At the other end of the scale, if you're buying a car from a dealer for 150BTC, you will have already filled in a load of forms with your name and address, and probably shown ID, so the dealer will also probably be happy seeing you leave with zero confirmations.

Other things to point out - you could easily buy a car with a credit card, then call up your bank and say that you didn't make that purchase, and they'll cancel the transaction. You'll probably go to jail, but why wouldn't you also go to jail if you did the equivalent with Bitcoin?
We could also just use small-denomination private keys as hard currency (physical Bitcoins) for buying small things like coffee.