I don't know if there are any special tools they use, but to me, it's very easy to know who is using VPN in some cases. If you notice a particular user who, each time they login to their account, appears in a new location with a different IP address from their last login, is that not an obvious sign of VPN usage?
What if this user just travels a lot and gambles in between? 
Since the gamblers detected (by casino) were using VPN, VPN providers should have long ago offered a static IP address service from the same location (or did they already?).
And they can also confirm their suspicions by asking the user to provide KYC verification documents, which clearly show their country of residence, which will be different from the IP address the VPN will give to the gambling site.
Firstly, I hate KYC, especially if it is required by the casino. Secondly, why can't the IP address be different from the country of residence? A lot of people travel to other countries, and also live in other countries for years. In this case, the IP address will most likely be different from the country indicated in your KYC documents.