African Credit Scoring Challenge
'%3e%3cpath fill='%23001489' d='M0 0v6h9V0z'/%3e%3cpath fill='%23e03c31' d='M0 0v3h9V0z'/%3e%3cg stroke='%23fff' stroke-width='2'%3e%3cpath id='d' d='m0 0 4.5 3L0 6m4.5-3H9'/%3e%3cuse xlink:href='%23b' stroke='%23ffb81c' clip-path='url(%23c)'/%3e%3c/g%3e%3cuse xlink:href='%23d' fill='none' stroke='%23007749' stroke-width='1.2'/%3e%3c/g%3e%3c/svg%3e){kind=small}
Dear Zindians,
Thank you for your valuable feedback on the new feature!
To address the issues that arose in this competition, if a user selected submissions during the 15-minute window, their submission selections were reverted to what they were before the private leaderboard was revealed. We confirm that if you did not select a submission, your leaderboard score reflects the best private score from your two highest public scores.
For future challenges we have fixed the issue so that the submissions close on the hour stipulated under "Timeline". You will then have 15 minutes to complete your submissions selection and only after the 15 minutes have passed will the private leaderboard and scores be revealed.
Zindi is committed to fairness, thank you to raising this concern and aligning with our values.
The current leaderboard is now final and reflects a fair outcome.
Onwards and upwards!
Excellent and thanks.
Clear, thanks
.
Hi @Amy_Bray, I would like to confirm whether solutions that set the response to 1 for GHANA for all records are in fact valid entries and aligned with the competition rules?
This would be a type of leaderboard hacking where a post processing step was used to optimise competition results but is in reality not a real world solution with real world benefit.
Taking guidance from this post from @AJoel, in particular, the below note, I don't see how the solutions were permitted. Please can you provide guidance?:
"Zindi is committed to providing solutions of value to our clients and partners. To this end, we reserve the right to disqualify your submission on the grounds of usability or value. This includes but is not limited to the use of data leaks or any other practices that we deem to compromise the inherent value of your solution.""
I think the leaderboard is not yet sealed. What has been corrected is point no.2 of this discussion. Winners selection - Zindi. For point no.1, i think there is abit of work for the team to check code submissions to ensure usable solution in real world
Valid point. Looking forward to feedback on the threads point 1 when @Amy_Bray and Zindi reviews the solutions for the areas highlighted by @sparked
I think if we keep doing this, people will be deterred from bringing out their solutions on board which destroys the whole purpose of even having a community in the first place and I don't think Zindi is all about just competing. I prefer you leave it up to Zindi to decide. Congratulations to all the winners and I hope you do win!
@Amy_Bray I would love to suggest something. Wouldn't taking the best private score be the right option? I feel since we do not know what is on the private board until the competition ends, it would be fair to just take the best private score and give an opportunity for each participant to choose which one he or she wants during the 15 minute time frame. I am quite new to this, but how about someone overfits to the public leaderboard on his best public scores and therefore does not perform better in the private board? This will be a serious setback especially if the person is at an unfavourable time zone.
I was moved from 155 to 288 because my best public scores didn't really give the best private scores even though I can vividly code out the result that gave the best private score. I hope this is considered because it might disturb in other competitions especially if the person is at an unfavourable time zone. Thank you!
Hi CodeJoe, thank you for the question. This is a feature on Zindi that we have thought long and hard about.
1. We need to have solutions that are reproducible, my users selecting submissions they should have the code that reproduces that result.
2. We hope to encourage users to do version control and experiment tracking.
3. We need a generalisable solution. By letting you choose your private LB score could lead to probing if you make enough submissions. If you knew your private LB score, it would be similiar to you optimising a CV score, which would not be useful to client.
This article could be helpful: https://zindi.africa/discussions/14812
@Amy_Bray Well said, thank you for the vivid explanation. I will take it into consideration.
Bonjour non mon score actuel n'est pas mon meilleur score privé veuillez le corriger