I have not thought it through, wanted to have feedback! 
From my background the intuitive thing is to use F1 measure as a score. Or area under ROC curve. There are good wikipedia articles about these.
These seemed to be concerned with binary outcomes. I want a metric that is continuous. I am developing one currently