Task Mate Kenyan Sign Language Classification Challenge
Can you classify words in Kenyan Sign Language?
Prize
$5 000 USD
Time
Ended 6 months ago
Participants
177 active · 503 enrolled
Intermediate
Everyone can hide his score
Platform · 24 Nov 2021, 15:51 · edited ~10 hours later · 5

I was able to score 0.1523 which made me in the top 10 (8th). The problem is that when I select a low score submission in the "Submissions" section, Zindi's algorithms show the best-selected score and not the best-submitted score. This is not how it is supposed to work. In fact, what we can see now in the LB are fake scores and people can reveal their real scores just before 1 minute of the end of the competition. The leaderboard is unfortunately useless.

The score of every user should be measured this way:

  • Public_score[user(j)] = FUNCTION([score(j)(i)["score"] for i in number_of submissions[user(j)]])
  • Private_score[user(j)] = FUNCTION([score(j)(i)["score"] if score(j)(i)["is_selected"] for i in number_of submissions[user(j)]])

  • Where user is the list of all the competitors in a competition; and is a list of
  • j is the index of a user
  • score is a dict that contains all the user_ids and every score(j) contains ALL the scores submitted from a the user(j) under the format {"score": int, "is_selected": bool}
  • number_of submissions[user(j)] = len(score(j)]
  • FUNCTION is the max or min function that depends on the metric of a competition (For example, min is the function we need to use in this competition)

This simply means:

  • For public LB: rank the users based on the best submitted score
  • For private LB: rank the users based on the best selected score.

I just gave you the solution and it's up to you to fix this bug and implement the simple algorithm I wrote.

It has been 2 years since I decided to leave Zindi (stop competing regularly) because of those issues:

  • Bad choice of the metric (like in this competition, what is best? log loss or accuracy?)
  • Wrong implementation of the metric
  • Unlimited submissions per day
  • Data leaks

Zindi is born to make the African AI community better, but this is not what it is doing right now. People just come here to earn some extra money and that's it. I saw some Kaggle Grandmasters coming here and compete but no one of them share his solution or approach. Do they do the same in Kaggle? of course not. They are here to earn easy money and leave.

Please, listen to our feedback and make @ZINDI a better place where people can interact, learn and share.

Discussion 5 answers

I agree with everything you said. @ZINDI please fix this.

Thanks @ragnarok bringing Up this Topic. @ZINDI please fix it asap!

Thanks for your raising this concern. I planed to post a similar thread as well but I don't think Zindi has human capacity to fix this issue in the near future.

very important issue, and people will not like @ZINDI because of such issues. I think this leads to a very bad experience as well. Earlier I was a regular Zindi competitor too, but such issues are making me thinking of devoting my time to a better place. This issue was also addressed to Zindi privately many times, but Zindi thought of this bug as a great feature :D. I hope Zindi fixes this issue ASAP

Hi Ragnorok,

Thank you for your message and the thought that has gone into this.

We have raised it with the dev team and it will be put on the backlog.

You will not see a change soon but it is on the backlog and will be prioritised accordingly.