As long as you don't lend money to someone to gamble, you don't have to be afraid when someone else invites you to gamble with that kind of money. Because the person who has to return the money is the borrower, not you unless you have made a promise to return the money together when you lose at gambling. But if you are only invited and are not told about it by the person who invited you, I think you are free from anything so you don't have to bear paying the money.
You should not be held accountable for something that you know nothing about, if your friend borrows money to gamble and invites you, then he is looking for someone to accompany him to the casinos and not someone to help him pay back the loan when he loses it. It could be you are going to participate in sharing the profits that comes out from the bets but the loss should not be your concerns as you do not join him in borrowing the money. If he also decided not to give you some share of the profits that you made from gambling, you should not be angry but that act should tell you the type of individual you are friends with. If you find out that he borrowed the money, as a good friend you should advice him to not use the money in gambling but return it and only use spare money for gambling incase it gets lost.