You need somewhere to exchange the wallets for money. Those to whom you transfer, too, will need to find a place to exchange the bitcoins for cash.
I know this isn't the case all the time as we do have shops and services that accept Bitcoin in direct exchange for goods and services without converting to fiat.
Well i guess it isn't only the reason why people avoid bitcoin, i guess people who avoid bitcoin does not have any idea what bitcoin is. And maybe those people who refused bitcoin though they already have idea about what is bitcoin is that they are being afraid of taking risks and maybe not willing to learn more about it. And yes we do have shops and services already that accept bitcoin as a mode of payment.