It's not anyone's business whether you gamble or not, unless you have a family and you have to feed your kids and care about them in many other ways, like buying books for them, smartphones and stuff. First provide them with everything they need and only then you can gamble because it is likely that you will lose money and losing to gambling is not a good excuse for not buying for your kids what they need.
Of course, when you are able to take care of your family responsibilities from month to month, that would not be a problem when you are gambling and it is not anyone's business. It shows that you are matured enough to control yourself and manage the little resources in your possession to live a responsible life. It is when you make gambling your priority and shy away from your family responsibilities, that it will bring insults and disgrace to you, because you have given room to every Dick and Harry to advice you and say nonsense to you.
There are really just those people who are really that introverts on which they dont really like for someone to have to say against them or would be assuming that they are really that being talked by other people around them and this is why they would really be trying out their best on suppressing on whatever the activities that they are dealing on with specially with gambling on which this one is really that having that negative views and impressions in the community. Just like on what you have said that as long you are able to do the responsibilities on which you must do then this what matter the most or something which is really that important. Dont mind if you are really that having those people around saying about you but its true that its none others business on how the hell you would be making use of your money on any means.
You are the ones who do earn it and you do have the full rights on what you should do from it.