The traditional crowdsourced machine learning tournament depends on a holdout dataset. The holdout data is some historical data known to the tournament organizer and unknown to the data scientists participating in the tournament.