I do agree with you, when I shop using debit/credit card (because the merchant doesn't accepts Bitcoin), I am always worried that my credit/debit card information might get stolen and someone might steal the money from my account or shop using my credit card. But when paying with Bitcoin I don't have to worry about these things, because no one other than me can know my private key and even if they did, by the time they do any damage, my funds would have moved from there to another address. So A.) Its more Secure.

B.) I am in control of my funds, banks can't seize my funds for whatever reason.
C.) I don't have to pay huge fees to banks for holding and spending my own funds.

Those are some of the reasons I prefer shopping with BTC but OP if you don't feel like using it, no one is forcing you to.