AI/ML for 5G-Energy Consumption Modelling by ITU AI/ML in 5G Challenge
I'm thinking about point 3 from previous post, that the users who did not make a submission selection, should not assesment\present in leaderboard or should assessed by maximum score.
Many other participants who made submission selection had better scores in their other hundreds submissions. Therefore, the leaderboard should be scored on the minimum score basis from all submissions to ensure fairness and accurate representation of performance.
In this case participants from point 3 would be in equal conditions with the others ..... otherwise they got the advantage over others.
This is indirectly confirmed by the fact that some of the participants’ submissions were made 2 months ago. Therefore, maybe the best submissions were selected among all the results for these users, and according to the rules, the participant must himself determine 2 variants.
Hi Pavel,
As mentioned by Julius in the other thread, and lacking in my explanation.
For users who did not do submission selection, we only look at their public score, the bug is that it was choosing their worst public score, instead of their best public score. The correlating private score is revealed.
Oftentimes, the best public score is not the best private score.
Our rules recommend you do model selection for a couple of reasons, but users are welcome not to do submission selection.
The reasons to do model selection.
For this particular competition, users in the top 20, will be required to submit a report and prizes will be awarded on:
P.S. Emails to top 20 should be sent out soon, please do not worry, we have not forgotten you.
Thanks for this complete explanation. Reproducibility of the result is indeed a separate big problem (for example, Catboost on the GPU, or other libraries when processing features, not to mention versioning :))).
Consider mandatory model selection in future competitions to bring it closer to real-world development conditions.
When you have to choose in any case which version to deploy based on a holdout sample (like 50% public in a competition) and then your model face the new data (like other 50% private in a competition).
Anyway, lets the best model wins!
Thanks.