Gambling depends entirely on the individual and everyone likes to play freely. As far as I'm concerned if you can keep yourself disciplined while gambling then you don't need to worry about what people say about gambling. Those who cannot maintain self discipline and gamble irresponsibly will be adversely affected by gambling. If you keep yourself under control no one will know about gambling.
If you keep a secret as well as you can, you will definitely find out. Your neighbor is even smarter than just a detective. even the slightest bad thing will definitely be known, especially neighbors who don't like you.
so just let what they say as long as it doesn't affect their lives. Do what you usually do, in this case gamble, so keep gambling with your own money. They also won't know what good gambling is like. they can only ridicule and look bad at gamblers