As long as you are still sufficient for your family's needs, there is nothing wrong if you gamble. Enough first your family needs, and the rest you can fight at gambling, with that you will play without pressure and maybe luck will side with you.
I agree with you, it is not wrong if you do not do it regularly. Also if you are discipline to do it, you know your limit, you do not gamble all your money. You make sure that you do it just fun but not for looking money to gain to.
We can consider it wrong when it results negatives in your life and you bet all your everything just to provide your gambling.
It's alright if you do it regularly as long as you have the discipline so you can control your time and the amount you risk.
Always think that gambling does not give us profit, instead, it's only a game that could give us some kind of entertainment where we have to spend only what we can afford to lose, that is the golden rule that we should follow.