A lack of knowledge about even the existence of Bitcoin keeps a majority of consumers away. For the rest of us - there are some who know but don't bother simply because it's more effort to buy Bitcoins and then spend them than just use your credit card to buy whatever you want to buy.
From the rest of us who dont bother buying bitcoins for spending them maybe doesnt know that if we buy and hold them for a while they will grow with the time maybe that is why we dont bother buying it, if we come to know about that or any other factor of bitcoin we would definitely bother to buy bitcoins if we can afford.