That's what they believe it will do to your life so they judge that gambling is bad and it's a sin to do it.
It's an old school judgment for me, gambling is never a sin as long as you know how to control yourself, they need to understand and be educated about gambling, but since they judge without knowing it, that means this kind of people are close minded and it's hard to explain to them and they will continue to judge you.

Just leave the town or hide your gambling activity, nowadays we can do it online, so it won't be much of a problem, you can still mingle with them, without them knowing that you are a gambler, lol...