Task Mate Kenyan Sign Language Classification Challenge
Can you classify words in Kenyan Sign Language?
Prize
$5 000 USD
Time
Ended 6 months ago
Participants
177 active · 503 enrolled
Intermediate
Evaluation metric
Data · 21 Jan 2022, 20:45 · edited less than a minute later · 1

Hello,

Train dataset in not clean, there are many wrong labeled images, also some images blurred. If the test dataset has the same issue, the Log Loss heavily penalizes. So, is Log Loss is the best metric in this competition? I think accuracy is nice metric, also it is interpretable.

Discussion 1 answer

I would have to agree. Accuracy is the better, fairer most sensible metric.

The objective should be to classify an image to a class. With log loss, we have to pay attention to the other (wrong) probabilities, which depend on the model, training, loss function used etc., so that they march whatever it is the marking scheme is.

If I predict the correct class e.g

0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0 ...

my log loss will be higher if the correct answer is

0.5, 0.6, 0.3, 0.4, 0.2, 0.9, 0.1, 0.2, 0.2

much as the argmax for both the logits is the same i.e. index 5

@zindi please reconsider. It is not too late