Africa Biomass Challenge
'%3e%3cpath fill='%23001489' d='M0 0v6h9V0z'/%3e%3cpath fill='%23e03c31' d='M0 0v3h9V0z'/%3e%3cg stroke='%23fff' stroke-width='2'%3e%3cpath id='d' d='m0 0 4.5 3L0 6m4.5-3H9'/%3e%3cuse xlink:href='%23b' stroke='%23ffb81c' clip-path='url(%23c)'/%3e%3c/g%3e%3cuse xlink:href='%23d' fill='none' stroke='%23007749' stroke-width='1.2'/%3e%3c/g%3e%3c/svg%3e){kind=small}
TLDR; if you wanted to cheat via data leakage, there is a very easy way of doing so.
Hey @Zindi and all,
Start with your current best model. Now continue to just change your first prediction; if you don't see any difference it is in the held out test set and so move on to the next. Manually change this prediction until its error is zero (or move on if again it is in the held out test set).
In this way, you can find all the true values of the labels at least in the public test set. Because there are only 90 test points, of which only around 30% are in the public test set, you could easily use 300 submissions to do so. (There are of course smarter / more efficient ways of going about this, but you get the gist)
How is @Zindi addressing this? I'm not sure how you could even ban someone from doing this, since changing the value you predict for a test point and seeing what it does to the public leaderboard is just another kind of model debugging technique you could use.
Great question! Smart6.
I am not too afraid of this issue as we always overfit the public LB in one way or another. However I am scare of 2 groups.
Group 1: People share ideas and data but not merge. There will be a shakeup in this competition so having "more final submissions" will increase your chance to be on top. 4 people in a team will have less chance to win compared to 4 different teams.
Group 2: People know how to select best submisssions for the private LB. Plato is an excellent example. We need to learn from this group.
I'm not talking about "somewhat" overfitting to the public LB one way or another, I'm saying you could reliably get a perfect score on the leaderboard within 300 submissions, which is much more problematic.
Unless of course @Zindi is fine with this?
Also, the group 1 and 2 argument applies to every competition. My criticism only applies to this particular one.
okie, if a team got a score of ZERO then what?
1. The private set is different from the public set so it is not a big deal.
2. I don't think Zindi allows them to use the labels directly in the code
PS: I knew (Zindi should know) people could probe this LB. In the worst case, 100 subs to know which rows are in the public set. With 200 subs / 30 rows == 7 sub / row, you could easily probe the LB.
Then we are in agreement :)
I also noticed that. I tried to submit with the test data and the result isn't perfect.
You dont have rerun in zindi platform so the LB Public/private are made by the submission file evaluation only.
If the "perfect submission" isnt good. It could mean that the sub part of the test dataset used for submission is somehow uncoherent or was modified.
The challenge could also be to properly use the test set to figure out the submission part.