People in general thinks that gambling is bad so even if you're not around high profiled people, you will be ashamed of gambling. It's the same thing as random sex, smoking weed, and other stuffs. You have to be around people that have the same interest or do the same things even occasionally for you to not get judged. Just avoid talking about gambling as much as possible and only talk about it when either you know the person you're talking to is into gambling or the topic just came naturally.
You can choose to keep silent about it well it depends on the situation as well. Some people might take it as a serious thing in some occasion while some will see it as a joke. Im sure that alot of people will see it as a negative thing so perhaps being silent about it is actually much better