Africa Biomass Challenge
Helping Côte d'Ivoire
$10 000 USD
Completed (almost 3 years ago)
Earth Observation
Prediction
1223 joined
276 active
Start
Jan 27, 23
Close
May 21, 23
Reveal
May 21, 23
Modifying the BIOMASS manually to overfit the LB score
Notebooks · 17 May 2023, 12:40 · 4
Does modifying the BIOMASS manually mean getting disqualified from the competition ?
I would like to know from those who have experience with ZINDI's competitions how strict ZINDI are with reproducibility ? Otherwise, if they are like some other platforms who accept even low quality solutions with no rep-check then it would be a loss of time for many of us, right ?
Hello @mohameddhiab
yeah ZINDI is usually strict about Manual Probing !
Don't worry, the private LB will be public LB^(-1).
All the best
So basically an ensemble of models gives me 80 RMSE while a manual submission with ONE constant value gives me 58 RMSE (If I submit more I can easily get better RMSE)
I can see some people in the top 10 submitting more than 100 times, it is so clear they are trying to overfit the LB.
But I can think in a scenario where someone tries to overfit the LB just to get a position where you don't end up in a position that requires you to submit you code, but end up in a position that grants a Silver or Bronze medal and still earn some points. I know that it's very hard and expensive to the host to label the datasets, but small test competitions are vulnerable to this kind of "cheating".
Yes but what @assazzin is trying to say (hopefully i understood correctly) are that the distribution of public test is so different from the one used in private test. (Which is by the way bad, as we want the LB to be slightly correlated to the private score)