It depends on the address, In the case of "XRP" currency. The reason why the exchange knows the address is coming from gambling activity because the address is already have a custom name "STAKE" so they are really easy to be recognized.
The only addresses that will be difficult to know are noncustodial wallet addresses, if you make use of binance, coinbase or any other exchange to send coin to me, I can track it to the exchange most especially if the address has been used countless numbers of time. I do not know how gambling site work, maybe like those exchanges, this will make it easy to carry out some searches with the hell if some sites that will reveal if the address belong to a gambling site. The best is to use noncustodial wallet.