AirQo Ugandan Air Quality Forecast Challenge
$5,000 USD
Predict future air quality levels and empower communities plan and protect their health
263 data scientists enrolled, 88 on the leaderboard
14 March—31 May
Ends in 2 months
Selection of Final Submission - Help Team Zindi
published 21 Mar 2020, 06:13

I am new to zindi platform, and hence I am not sure about how the final submission works here, which is used to evaluate the private LB.

Its mentioned in the rules that the best submission is used to evaluate the private Leaderboard. Now there can be 2 cases:

A. Best submission in the public leaderboard. This is unfair because later in the competition someone can get a model with better cross-val score but lower public LB score. And that competitor might want to go with the model with better CV score, so to avoid overfitting the public LB. But if you select best public LB as the final submission, this is not possible and it becomes unfair.

B. Best submission in the private leaderboard. This is again unfair, because more the number of submissions you make, more you increase your chances of getting to the top. Because you have so many different submissions, the probability of one of them going to the top increases. It may or may not be the model you are confident about, just a matter of luck.

Hence, I request team Zindi to allow competitors to chose their final submission, which would make this very fair. If any one has any other suggestions please discuss.

I think the current position from Zindi is fair. I say its fair because we all have equal submission numbers per day. More so, they have reduced the split of public leaderboard. This reduction in my view is a great move because the public leaderboard is now given less weight. This will force competitors to build a robust model or face the risk of going down the rank upon shakedown.

But I should get the chance to select my final submission, as later in the competition I might build a more robust model, which scores less in public LB, but I can see it gets a better CV score. That model is my best model, but I can't select

I believe the final score that is automatically chosen by Zindi at the end of the competition is the best score on private LB which translates to the model that was more robust.

Yes even I am confused about this rule. But what other people have told is that best public leaderboard score is selected. Please confirm if you are

The best private score is selected not public

Thanks a lot. This clears my doubt. I agree, people need to build more robust models and not rely on public leaderboard. Thank you.