we are supposed to predict for week 48 and 49 at this point. Meaning the rmse and mae values we are expected to see on the lb after this week 49 has ended is for both the 48 and 49 weeks. Why are we given this information now? shouldnt it come after the 49th week has ended or am i missing something ?
because people can just fill week 48 data with ground truth and get perfect scores. or maybe we are only supposed to be evaluated on one week? in this case week 49 now that the gt for 48 is out?
Hi @Koleshjr, thanks for raising this — it’s a very valid concern.
You’re right that ideally we should have updated the leaderboard for both Weeks 48 and 49 together, once the true values for both weeks were available. Sharing Week 48 scores alone is not good.
Here is why:
In Week n, participants submit predictions for Weeks n+1 and n+2.
W47 → predicts W48 & W49
W48 → true values for W48 are released
W49 → predicts W50 & W51, and now the true values for both W48 & W49 are available→ This is the moment where the leaderboard should have been rescored for both weeks together.
After that:
W51 → true values for W50 released
W52 → predicts W01 & W02
W52 → true values for W50 and W51 are released, so the leaderboard is updated again
Competition closes
After the competition ends, participants can no longer submit new predictions.
W03 → true values for W01 & W02 become available,
and final scoring is done and the private leaderboard is revealed.
Because the Week 48 values were already public, the perfect approach would have been to wait for Week 49 as well and then score Weeks 48 and 49 together. Unfortunately, that window has passed, so our focus now will be ensuring that the rest of the evaluation process follows the intended sequence smoothly.
we are supposed to predict for week 48 and 49 at this point. Meaning the rmse and mae values we are expected to see on the lb after this week 49 has ended is for both the 48 and 49 weeks. Why are we given this information now? shouldnt it come after the 49th week has ended or am i missing something ?
because people can just fill week 48 data with ground truth and get perfect scores. or maybe we are only supposed to be evaluated on one week? in this case week 49 now that the gt for 48 is out?
Hi @Koleshjr, thanks for raising this — it’s a very valid concern. You’re right that ideally we should have updated the leaderboard for both Weeks 48 and 49 together, once the true values for both weeks were available. Sharing Week 48 scores alone is not good. Here is why:
After that:
After the competition ends, participants can no longer submit new predictions.
Because the Week 48 values were already public, the perfect approach would have been to wait for Week 49 as well and then score Weeks 48 and 49 together. Unfortunately, that window has passed, so our focus now will be ensuring that the rest of the evaluation process follows the intended sequence smoothly.
Thanks @AJoel for the quick feedback.