Somebosy tried a model submitting new_turlte all the time and it scored 0.01360544217687075 and he deduced that there 2 / 0.01360544217687075 = 147 images scored on the current leaderbord. This represents 20% of test data. Hope it helps.

You're right. I also think that the 20% estimate of pulic LB is probably wrong and this is to be lower. For me i validate on roughly 600 images per fold and train & val score are close. That's why i suspect that the percentage of images involved in public LB is even lower maybe less than 5%. But if this not the case, we should seriously worry, because private LB could be lotery.

For sure there is something strange in LB scores (I tried to tell zindi, but they say that is OK). I think the best option is trust CV (make submissions doesnt make sense).

I think you're doing good. All you have to do, is to trust CV as long as your cross-validation is ok.

One thing that maight help is to keep track of other metrics that might be 'harder' in conjunction with the Apk, such as macro accuracy and the sorts. I've seen the apk go up in validation yet these other metrics fail, and models that have done better also in the other metrics tend to do better on the leaderboard

Hi @astenuz what is apk ?

It's the average precision at k, I think that's the one we're using to score entries right?

Okay thks

Is there any new class in the test data that is not present in the training dataset? I guess new class data are assigned to `new_turtle` but there might be more classes. I hope I'm wrong.