The reason was simple, religious people believed gambling is owned by devils.
I can't remember anywhere in the bible that said gambling is of the devil. And there are no religious groups that have included gambling as a sin their doctrine. What i know is that the religious people consider gambling as an irresponsible habit and such habits is one that Christ do not support as a good behavior. There is a popular story in the Bible were traders, and gamblers and so many other stuffs were taking place in the temple of God and Jesus came and chased everywhere out destroying what they came with. He was angry that this kind of activities should be done outside the temple and not within the temple. Which means gambling can be done anywhere else outside the temple and not to be considered as a sin.
Maybe yes, in your religion gambling is not so harshly forbidden, because as you say, I infer that gambling can be done as long as it is not done in your temple or place of worship, but perhaps you know more about that than I do.
But for Muslims gambling is forbidden and a sin because it is risking money on something that is harmful, and indeed some of the teachings are that it is a game of Satan or the devil although it is not written in our books, but the translators and scholars say that gambling can bring destruction to a person, and indeed our prophet forbade such activities even though there was no gambling then as there is now but the mechanism is similar, so that is the conclusion taken about the view on gambling.
This will be a dilemma for many Muslims who like gambling, but many are also against it and prefer to hide their gambling activities.