Well, I don't see gambling as the most expensive fun activity when you can spend $5 or less to gamble. So, it's not a must make you rich activity neither. If you are already rich and you are gambling for entertainment, that's a different case, so many people are gambling to make a living and they are not rich, but I guess it's easy for a rich person to say they want to take gambling as an entertainment but poor people would definitely want to make money from it and not to throw away money like some rich will do.
Maybe there would be a lot to talk about if it was in the context where I spend too little on gambling, like you mentioned for example 5 dollars. And everything else that I could potentially spend on gambling I just set aside, and then the money I have saved I would invest somewhere, in some assets, or simply buy Bitcoin. In that case you could say that I would be growing wealthier. But if my budget is too small then simply not playing gambling games definitely will not make me richer. It would just be small savings that I could spend on some necessary things.