Two colluding partners is difficult to maintain (since either partner can gain an advantage by first agreeing to collude and then operating outside the collusion).
More colluding partners is even more difficult to maintain. There is more opportunity for a partner to operate outside the collusion, and more opportunity for the secret to leak.
Even if you could get 10 pools to collude without cheating on each other, there's nothing stopping me or anyone else from starting up another pool.
point is with even 1 pool the network would be relatively safe as the costs for attack would be significant for miners (who could just switch to another pool they make if it's the operator causing issues)
either way miners answer to the users via the market or they lose money they can't recover