Thank you for the response. The issue I had with the column is that you should not have information in the test dataset that will not be available when deploying the model. Its ok to have the damage column in the training data but one can use it to improve their position in the leaderboard without fitting a better model by simply setting all test predictions to zero whenever damage is not "DR"

the concern is: Why is the column "type of damage" available in test data? Will it be available in private test data too?

I am new to Zindi but my understanding is that the private leaderboard is just a subset of the test data they've given us? So that would mean "type of damage" is available at least until they test the solutions themselves. However, I assume the organisers don't want that. It might be useful to know if they will now assess the submissions any differently to originally planned; it says they would assess the top 10, but the top 10 may be full of solutions with data leakage issues.

Thanks for the comments here - we have organized the organizers to take the "type of damage" column out, since it is not supposed to be used in the solution. We had not realized that it would be a cause for confusion, hopefully by taking that column out, the issue will be resolved!

sorry, but the action is still unclear to me. Will we have a new testset (without column "type of damage") or will we continue with the current testset (which has already leaked)? Continuing to compete with the recent test seems a bad idea since many competitors can leverage the leak to get high scores but it is totally useless in real-world application.

Solutions that use the "type of damage" column will not be accepted. You can continue, but please don't use that column in your solution. Thanks!

@berber-kramer Thank you for this clarification. However, it is still challenging for us to monitor our progress because we are unsure if the Leaderboard (LB) is reliable or not. This uncertainty might lead us to prematurely abandon our efforts, fearing that we are performing poorly on the Leaderboard, while the top-ranked participants might achieve their scores due to factors related to the 'damage' column.

If it's possible to replace the current datasets excluding the 'damage' column and assign new IDs to the images, it would help establish a more trustworthy Leaderboard. This way, we can better assess the effectiveness of our approaches in comparison to other's scores.

"Solutions that use the "type of damage" column will not be accepted". But we can use this column when training the model, right?

This is something for the organizers to comment on; I'll forward your question, but please also reach out to them directly.

Yes, just don't use these data from the test dataset :)

Why do you say that `extent` for non-DR is set to zero?

Below I displayed a histogram for WD (weed) and it clearly shows that there are non-zeros values. And I'm confused.

1. Does it mean that an example with an extent > 0 "WD" should be "DR"?

2. Or I should change value for `WD` to zeros?

But as I can't use `damage` column, the second option is not permitted.

I just want to understand if we should predict `extend` for `non-DR` damage (like DR would have values >0, non-DR would be always 0).

Running your command creates a histogram where all values are zero. You might be missing something eg plt.show()

Thanks for checking this out. Look like I checked the file, which was downloaded together with images, from S3 bucket. And in that file, values for "WD" are not zero-out.

Just to make sure:

Now I run this for the Train.csv and get:

import pandas as pd

pd.read_csv("csv_files/Train.csv")

for damage_type in list(df["damage"].unique()):

    extent_per_cat = df[df["damage"] == damage_type]["extent"].value_counts().index

    print(f"Unique extents for {damage_type} damage type: {extent_per_cat}")

and get:

Unique extents for WD damage type: Index([0], dtype='int64', name='extent')

Unique extents for G damage type: Index([0], dtype='int64', name='extent')

Unique extents for DR damage type: Index([10, 30, 40, 20, 50, 60, 90, 80, 70, 100, 0], dtype='int64', name='extent')

Unique extents for ND damage type: Index([0], dtype='int64', name='extent')

Unique extents for DS damage type: Index([0], dtype='int64', name='extent')

Unique extents for PS damage type: Index([0], dtype='int64', name='extent')

Unique extents for WN damage type: Index([0], dtype='int64', name='extent')

Unique extents for FD damage type: Index([0], dtype='int64', name='extent')

So the non-zero extent is set only for the DR damage type