One who wins makes more and one who loses - loses more. I am not much into gambling but yeah I played some games I can say that I saw losses more than wins.
It has nothing to do with being poor or rich, it is all about luck (IMO). The profit of rich gamblers can be seen by the poor and they get attracted to it but no one cares about the poor's loss/profit.