If the gamblers are primarily buying altcoins with bitcoin, then I don't see how that's bad for Bitcoin. They couldn't hurt the price anymore than they helped it in the first place. It's not really any different than using bitcoin to gamble dice or poker. Later updates of the Bitcoin client might even adopt some of the good alt-coin ideas. It's when they bypass bitcoin and have altcoin-to-fiat exchanges that it could be a dangerous distraction. Money is the most marketable commodity. Like the Highlander said, "There can be only one."