Even though our neighbors know about our gambling activities, we don't need to think about it, especially if we can maintain good discipline. We also don't need to listen to what they say about what we do. They will never know that we can use gambling as entertainment.
Sorry but I think it's not something easy to do. You can ignore your neighbors who know about your gambling activities, but you cannot ignore the negative impacts that result. Your neighbors will keep telling bad things about you. Even though you only gamble for fun, your neighbors will see that you are a gambling addict who is wasting your family's money. I personally care a lot about what people say about my gambling if they actually know the facts. This is different from those who just guess you are a gambler. I wouldn't care even if he told the truth about me.