Is this normal or am I doing it wrong?
Depends on the niche and the audience. If you sell computers, gadgets, other tools, web services etc., then it's reasonable to have customers willing to pay in bitcoin. Oppositely, if you're oriented towards people who may not have an idea of what's bitcoin, then they will simply not pay in bitcoin.
Every time that I've offered Bitcoin as the only payment method available I've ended up losing the customer.
But, you shouldn't force them pay in bitcoin if they don't want to. If a person doesn't know what's bitcoin and is forced to use it, he'll probably just choose another store.