People buy goods and services face to face billions of times a day around the world, and the vast majority of transactions proceed without issue.
A merchant and a customer, which is what you're referring to, don't have the same relationship with a taker and a maker of a trade. Building a reputation is a must for the merchant, because that's what usually draws the customer. It's safer to buy marihuana from a reputable local store legally than from a drug mule illegally.
Same goes for peer-to-peer trading with a complete stranger. There's been a case in my local board wherein the OP got ripped off that way. The taker just took the cash and left.