The same reasoning can also be applied to credit/debit cards, leather wallets, and other "containers" of money: people who have or use them in public are clearly not that poor. Yet, we rarely see people being attacked just after paying at the checkout of a grocery store.
Because they have grown to be very common in modern society. Especially the usage of Credit Cards today, they're more viewed as something that's holding debt to the banks than a status symbol. Perhaps 70 years ago when the first Credit Cards were issued to important clients, they were good targets for robberies. Currently, no. Credit Card holders usually have more debt than wealth.
That doesn't really matter because credit card fraud is still a very real thing and they will not only drain all the money in your cards but also max out your credit limit as well.