Cain’s Jawbone Murder Mystery
Can you solve the 6 murders of 1934?
Prize
$300 USD
Time
Ended 26 days ago
Participants
39 active · 223 enrolled
Intermediate
Prediction
Media
Hardcoded solutions clarification
Connect · 30 Dec 2022, 13:02 · 8

I remember times, when people got banned on Zindi for usage data leaks, for image competitions it is still against the rules to use image metadata...

In Cain challenge we see extensive LB probing, more than 5 ground truth pages were leaked on the forum (current rules for Cain asked us to not share even our codes or our general strategy).

There is part of Zindi's rules:

To this end, we reserve the right to disqualify your submission on the grounds of usability or value. This includes but is not limited to the use of data leaks or any other practices that we deem to compromise the inherent value of your solution.

So I believe hardcoding values from LB probed results is violating the rules and the spirit of Zindi.

Still, we haven't recieved any reaction from Zindi and today is the pre-final day of the competition. People use leaked values to jump at the top of the leaderboard and, in my view, final result is a complete LB carnage.

I am guilty of using probing too (it is not against the rules), but used points for model-guidance and tuning. I thought it would be insane to hardcode values for final submission but now I am not sure that Zindi has something against it.

@Zindi, please clarify things.

Thanks in advance, happy New Year every one.

Discussion 8 answers

Hello,

I understand your frustration when you could not profit from the LB probing. However, noted that if a solution is purely based on the public LB, that solution will have a bad surprise in the end.

The purpose of the LB is to motivate people to submit their solutions and have a quick feedback. Otherwise, we don't need the LB, just the train set. If you also follow the kaggle competitions, they even have a prize for people that are on top during the development (https://www.kaggle.com/competitions/santa-2022 is one example).

For this special competition, it is more for fun rather than a real ML. People have tried to solve this problem manually but so far (in 100 years), only 4 people managed to solved it.

I believe this competition's objectives are much more broader than just some ML-but-just-better-than-random solutions.

30 Dec 2022, 13:54
Upvotes 1

Rules are rules. It is consistent thing.

If this comp is a special case this fact should be reflected in the rules for this particular competition.

However, noted that if a solution is purely based on the public LB, that solution will have a bad surprise in the end.

True in general, but not the case here: private covers 100% of data. In addition, you still have an edge: for 10-20 pages you know right numbers, so you can leverage it to have better private score because you have less options of possible pages for private part.

Hello,

- "Rules are rules. It is consistent thing."

I do agree. Could you kindly point out the rules that we are not allowed to probe the LB ?

- "True in general, but not the case here: private covers 100% of data."

How do you know if the private is 100%. I think it is only 25 hidden pages.

- "for 10-20 pages you know right numbers"

We only know 2+7 public pages, which were already shared in the forum. Note that we had a solution with a score 0.06 (3 out of 50) before probing. But I am not sure if the score will be the same for the private LB.

1. I haven't said that LB probing is not allowed. I said hardcoding values from LB-probed results is violating the rules. It is literally in the title of this discussion.

2. Check the rules tab of this comp: Upon close of the competition, the Private Leaderboard, which covers the full 100% of the test dataset, will be made public and will constitute the final ranking for the competition.

3. I it is not about you or your team, just general showed a general concept of winning if hardcoding your submission is allowed.

I don't really want to discuss anything or to debate. Just want confirmation from @Zindi team.

Hi,

Description: "Your job, as the puzzle solver, is to try to put 75 pages of Cain’s Jawbone into its correct order, using AI NLP algorithms.

So, the winner will be with score = 0.1, I think :)

30 Dec 2022, 17:29
Upvotes 0

Oh, given that score, could we call it a solution ? "10% of pages are correctly sorted but 90% of pages are randomly shuffled". Not sure if any reader is going to read that sorted book :-)

Wait a bit @victorkras2008 you sort of let this cat out the bag. Now you can not reverse and argue against using probe? For this very difficult problem, unless you spend years on it, you have to probe. Question is more when to stop, so I agree with Nikita's question.

@NikitaChurkin I suppose zindi is on leave and we can debate this a lot. We can even agree between ourselves what is fair, but zindi will make final decision and as you say, it is not worth it to debate this too much.

You seem to take hardline stance - solution must be "pure" otherwise it is not "fair". It is easy to argue for or against it. I am not sure myself and wish zindi would comment, but I don't think it will in time. From my side, I think the answer lies in the middle somewhere ... and so while we wait we end up discussing and debating anyhow.

What makes the idea of "fair" tricky, is that in real life you'd use whatever you can in a good model. If you write a chess engine, you will load a known opening book into it. That is not cheat and without it, your engine will be weak. You will even load known endings into a strong engine and so a lot of the moves won't be entirely "pure". This comp is not just a middle game puzzle, it is a whole game and you need a bit of help to get going. It is really unrealistic to expect an AI to play this whole game.

Also, even if you don't use the probe directly, you can still use it indirectly to tune your model so probably even "pure" solutions are contaminated by probe, even if they don't show it. This is true for any comp with a public LB - your model development is guided by your public score and this is not wrong.

We can continue debating for a long time ... for me, I think, until zindi clarifies, I will at least use public probe and watch this and other discussions and adjust based on that. I will also discuss (more) freely with @Moto who has years of experience with this stuff so we have at least something we feel is fair, but, based on our past discussions, this is a fun and also impossibly difficult challenge, so actively using probe seems fine.

Not sure where to put this - we have the same discussion in three different places, but since it seems the conversation moved here, so here goes.

@victorkras2008 !!!! Wow Victor, as mamma said,

action speaks louder than words

You can say what you want, it is clear what you believe. But I am sure it feels nice, you are building the whole thing, from start to end-game. It is much more difficult, but much more rewarding. Look at score! Also, it is much more dangerous - perhaps you are thrown out for this. And with applied math, you have all the right tools for this holistic, integrated approach.

Perhaps we need to share probe results again? Probably not, but afterwards, once the dust settles, we must discuss this a bit more.