Can't agree more ! I think AUC metric will be appropriate for the Zimnat.

Agree. We should use AUC or F1-score

I think this is a good discussion to have. There are always upsides and downsides to any metric. I'll come back with some more thoughts later, but one thing I wanted to raise right now:

In your example, the 'good' model is making three predictions with 100% confidence that are wrong. log loss really penalizes confidence in wrong answers. If you take that same set of predictions but clip them, you suddenly get a much better log loss score:

pred = pred.clip(0.05, 0.95)

log_loss(true, pred)

> log loss of 0.14, muuuuuch better than 0.5s everywhere.

A good model, trained with the metric in mind, can output predictions that get a good score by taking the dynamics into account.

I have back-to-back meetings now but would love to chat more. From past contests where this has come up we've seen hat accuracy and log_loss are very closely related - generally a good model does well, a bad model less well. Obviously it's annoying if someone comes and does some metric hacking to get a better score without focusing on getting a better model, so I see that side.

I like the idea of AUC. Look forward to everyone else's inputs.

I agree with Nikhil's opinion here. Area under PR curve could also be considered for binary class classification problems. But still AUC/F1 score would be a better indicator of a strong classifier.

in my opinion , Nikhil is right but metric in competitions solely depends on the companies hosting the competition and what do they actually want to do with the metric involved,Ex Probably if the model goes wrong it's probably necessary just to check why it goes wrong in insurance it imperative that wrong values have to be penalized more than usual as there aren't any room for mistakes

I believe it really depends whether it's more important to predict one than zeros. But in my opinion, AUC would have been a much better metric.

However, the accuracy logic is flawed, this dataset is imbalanced, so you can't really compare accuracy with logloss or AUC and so on. Besides accuracy is a count based metric, where as AUC or log loss uses probability to compute the loss.

When your statement is somehow correct, I will ask not to be so categorical about the LogLoss metric. LogLoss foundamentaly forbid confidence in wrong opinions ! Imagin you're building a system that would predict if a person suffers form HIV, you will want your model to place wrong "ones" nowhere ! And, if it does, your metric must be rude to him !

Usually, the metric choice is up to the organizers and depends a lot in their use cases. Even if LogLoss everywhere is surely a bad idea, LogLoss nowhere isn't better !

I always share the same discussion in many competitions but no one is understanding what I am saying. PLEASE!!!!! LOGLOSS IS A STUPID METRIC!

I left Zindi for this reason

Thanks all - we'll take this up internally. I don't think we can change the metric on this one since it's launched and going, but for future comps/clients, we can push the clients to go for something like AUC or F1 where it makes sense.

I think @Sravan121 hit the nail on the head the point that in any problem like this there needs to be some humans thinking hard about what a model's outputs mean, and how the choice of metric affects that.

We shall see what the big bosses say but I suspect that looking at the list of top competitors here sharing ideas might be a good argument in favour of some new metrics to hack ;)

Thanks @devnikhilmishra for bringing this up, and everyone for all the input - it's really cool having such a large community of experts here on Zindi helping us learn :)

Agreed! I was in a Hackathon where average logloss of LB of all ppl was 0.73 or something.... Keeping all 0.5 i could get to 0.69 without any model... So... I support!!

good point Nikhil. +1

I would like to add 2 cents here.

In case of real life problems, the evaluation metric is always a big challenge. Because the business don't understand metrics like AUC and/or logloss. The goals are set keeping in mind improvements towards accuracy, recall and precision numbers. That being said, setting a hackathon eval metric is a challenge in itself. Here the idea is provide a best way to evaluate a large audience. A detail analysis is definately needed to set up evaluation metric for various hackathons.

As far as my experience goes, AUC works well with binary class classification problems. It is independent of threshold values and powerful in case of imbalanced data. Logloss is also a good metric but i feel it works better in case of multi-class classification problems.