yeah, of course, otherwise I would not know.

I am happy to share too. what's the vibe on zindi if I do a super late sharing of leak? on kaggle i will be downvoted to hell.

This is an example of a recent data leak discovered a few days before competition closed. The sentiments of participants are in the comments.

There are no down votes on zindi. It's good to share. Zindi team is also very fair too during evaluation of top solution

https://zindi.africa/competitions/microsoft-learn-location-mention-recognition-challenge/discussions/22497

Thank you so much!

No you'are not going to be downvoted on Kaggle. I think on kaggle, it would be aprreciated and they are quick to take action when the leak is confirmed. Generally, they extend competition deadline. I've seen it in more than 3 challenges.

yeah, it is possible. I realize I might need to eat my words about "knowly exploit"

no it is not in the image. Ok I'll share it later today.

Okay...

I apologize for what I said. But there is a way of knowing. I'll publish a function that boosts any submission. if a submission is not boosted, the leak is already in.

Hi @snow , I have watched the video and try to implement the function but I keep getting errors . Could you please share the link to the colab notebook used in the youtube video ???

I tested it, and it work and it's easy way to score !!!, but i think it must not be allowed.

I think what should not be allowed is the post processing step that is as a result of the leak no?

Btw what was your score before the leak? I have not tested mine yet as we are out of subs for the day

My bad

I've seen the video, I didn't even realize there was a pattern. Thank you, I think what you shared is very valuable as input for the competition organizers.

No worries bro :)

The score before the leak was 0.0024 and the score after were 0.0020. But it's was not the best score that was tested. And i understant that goal of this competition must be to give the raw probability of the model without any post processing.

I have a different opinion. I don't agree that the model picked up the pattern because there is no explicit encoding of location order in any column. If the model were truly capturing a pattern, it would require some form of sequential or positional information of the location, which isn’t present in our features.

Additionally, once you apply stratified K-fold cross-validation, any inherent order in the data is effectively removed, making it even less likely for the model to learn a spatial pattern unintentionally.

I don’t consider this a data leak unless:

1. Post-processing is applied to exploit the pattern.

2. The location order is explicitly encoded in one of the features.

This is similar to the case discussed here:

https://zindi.africa/competitions/geoai-challenge-for-agricultural-plastic-cover-mapping-with-satellite-imagery/discussions/23270, where participants used an ID column to boost scores. That was a clear example of the model leveraging an unintended pattern.

However, in our case, if I speak for our team, we are not encoding any location order. So, the key question remains: what exactly constitutes a leak? In my view, a leak only occurs if post-processing exploits a discovered pattern or if location order is directly embedded in the features.

food for thought,

If indeed the model learnt the sequential order of the location wouldn't it be worse on the test set since the test set is reversed??

Correct position

Trueeeeee!!!!!!!, Like the model picking up the leak itself doesn't make sense to me unless you explicitly introduce it as a feature. And also just like I had said before:

If indeed the model learnt the sequential order of the location wouldn't it be worse on the test set since the test set is reversed??

@Koleshjr Exactly, the argument contradicts itself

I agree with you @Muhamed_Tuo

I agree with your logic @Muhamed_Tuo , the exploit is in post-processing, not in model training, I think most would agree with that

That's what I wanted to say. Normalizing probabilies systematically lower Logloss It is just the mathematics of it The true issue, is the identification of flood location.

Now, may the jumping game starts :) . The LB is going feel different

it's going to be a wild 3 days. Hats of to you @snow for finding and disclosing the leak.

I know it's late to reply, but I don't see anything wrong with doing that, taking advantage of the fact that there are never more than 1 flood for one 'event_id'. However, I'm very skeptical that would actually improve your score; you could try that and see for yourself. Intuitively speaking, it's good to overshoot (giving a total probability of more than 1) for 'events' where you're confident you can pinpoint a narrow window where you think the flood must be, and just give up (giving all 0) for 'events' where you think any day is as likely as any other day to contain flood. A model would implicitly learn about this automatically.

Also, it's weird to forbid thresholding. In practice, it never works anyway (speaking from my experience with this competition and the recently-ended sepsis detection competition on Kaggle, both of which deal with extremely imbalanced labels), so why bother forbid it? Again, a model would be smart enough to adjust its scoring in order to minimize the loss; a naive improvement strategy such as thresholding its score can only mess up.

Same as you, I just tried the apply_leak function you shared in submission, and my score increased significantly. But of course for the final submission I did not choose the submission that contained the leak. There might be a shakeup in the private leaderboard.

Thank you for your share @snow