There are two problems with that I can think of:
1. Bank transfers can be reversed in some circumstances (account was stolen etc). Some days after Bob gets the money and releases the bitcoins, the bank may reverse the transaction.
2. I'd be worried that the bank may freeze my account if I receive cash transfers from someone who turns out to be a criminal or the bank freaks out for some reason.
I would much prefer a centralized exchange.
This concept could work if A and B meet up and payment is made in cash. This concept could also work for trading between cryptocurrencies (btc <--> ltc etc)
I think these are serious issues which need to be considered
These maybe hard questions to answer, but if #2 in particular starts happening for users of your system, you will become a VERY unpopular person, VERY quickly. Having a primary fiat bank account closed or locked is far more damaging and off putting for the novice user (who you've said this system would be for) than losing some fiat or some BTC.