For face to face transactions, confirmation time is one hurdle that i can think about.
Nah, it really isn't that big of a problem. If you're transacting really big amounts, you have to wait for at least one confirmation, yes. But if you're doing small amounts, you're fine just making sure the transaction isn't an obvious double spend. This only takes about 10 seconds. Trusted members of some website would be stupid to try and scam someone with an attempted double-spend!