You nailed it.
But many people do not care about it. They just want to have fun and money at the same time. I am not sure what goes on in the mind of horse owners. Do they really have soft heart? I doubt that.
Let alone horses, considering cock, spiders, and dogs, if they just want to make fun, they will make everyone fight just to make something to bet on. Maybe they cannot resist the old habits they have and don't care about the animals after all. When I saw the no-choice in the eyes of those animals, I translate them as cruelty and those people involved in this somehow don't have any hospitality towards those animals when they lose the game.
Romans used dogs, lions and tigers in their games and had a lot of fun watching either people slay these beasts or people being eaten. We crave blood and cruelty, that's why in many countries animal abuse still isn't punished. It's common especially in countries where animals are considered to be working machines that we're supposed to bend to our will and throw away, because god or whatever prophet said so. I'm against animal abuse of any kind, but there's a thin line between making a horse perform for the audience (jumping, parades) and racing. Should we ban all activities that involve animals, like dog agility contests and such?
As strange as it may sound, animals can die of natural causes. No one knows when or where it will happen, so consider that if an animal dies at a race, circus or any other place it does not mean that it died of cruelty.
I'm against cruelty, but you have to understand where the line is between cruelty to animals and quite natural. Personally, I think the Spanish carrida, cockfighting, dogfighting all fall under animal cruelty. But horse racing is an entirely different matter.