The ARC Challenge: Africa
Helping Africa
Flights to DLI
Challenge completed 3 months ago
Prediction
115 joined
37 active
Start
Jul 25, 24
Close
Jul 31, 25
Reveal
Jul 31, 25
How is Accuracy being measured?
Help · 26 Jul 2025, 06:52 · 4
I have made a number of submissions but I keep getting errors like: Missing entries for IDs test_0004_11, test_0004_12, test_0004_13, test_0004_14, and test_0004_15
Are my outputs' number of rows supposed to be the exact expected number of rows or they can be dynamic.
Yes, the sample submission already gives some insight into what the answers look like, including giving away the true number of rows for test_0000 (10 rows) since you have up to test_0000_10, and also gives away the number of expected columns (10), since each of those samples contains 0000000000, particularly for test_0000
The scoring metric is simple; we're expecting the sample submission format in terms of IDs. The target column 'row' is simply the answer in a string, so it's super easy for us to track the accuracy of the submitted row.
So assuming my model's matrix is smaller, should I fill in from the top left corner and leave the rest of the positions as zeros or should I make sure the model always produces the expected matrix size
unfortunately, accuracy is stringent. If the string (matrix row) is not exact, there's no plus for getting any part of it, so filling will be up to you.
Example if you have: "7777" and truth is "77778", the sample submission will be "00000", so you can already know you are wrong since we're expecting 5 numbers, the hard question is how do you know what to fill with?
I'll leave that to you as the pro.
Thanks