Makerere Passion Fruit Disease Detection Challenge
Helping Uganda
$1 000 USD
Completed (over 4 years ago)
Classification
Computer Vision
913 joined
171 active
Start
Aug 20, 21
Close
Nov 21, 21
Reveal
Nov 21, 21
Weird competition metric
2 Sep 2021, 18:55 · edited 2 minutes later · 8
If I submit a prediction of a model which was trained 30 epochs it will get a worse score than the same model that was only trained with 1 epoch. The less trained model predicted way more uncorrect boxes than the long trained one. So spamming with unprecise bboxes results in a higher score than predicting the fruits correctly. This doesn't make any sence. The metric should show a higher score at more precise predictions and not reward more predictions. @zindi
I totally agree with you, it's like that a worse model is better than a more robust one and it can be checked also visualizing the prediction on test set...I have posted a topic about that, because there are strong incoherences in the labeling, and I'm worried about the fairness of the test set...
Thanks @derInformatiker for bringing Up this Topic .
fruit_brownspot 733 fruit_healthy 664 fruit_woodiness 584 ?
Hate to break this lovely thread, but guys your models are overfitting -_-
Use a validation set to validate your predictions.
I don't why other competitors didn't intervene sooner 😂
overfit can be a problem, yep, but the Cross Validation of training set is not so informative...I think that there is some problem on labeling rules, for example what happens if a fruit of class 1 and 3 together? I tried some very stupid rules about scores and I get a better result respect to my CV...I don't know it's my fault (maybe) but this competition is very strange...
Whatever reason you say, the organizers would just chime in "that's how competitions are like" or "that's how data is like". anyways, your goal is to defeat other competitors which wouldn't matter since everyone is on an even playing field. Don't really expect your stuff to be in deployment - or whoever does use it in real-time, is a pretty big idiot.
all right...I don't like too much these kind of competition, but if this is the goal I will use my hacker's skills 😂...I'm joking...
just to be clear, I like a test set that is significative, that judge a better model than other, it's impossible that an efficientdet-d0/d1/d2 trained on trainining set, usinf classifiers and merge with an embeddings is worse than a basic faster rcnn trained on only 1 or 2 epoch, came on! if I show to you 100 inference of test set with an efficientdet-d2 you will say "wow"! and if I show you the same with the faster-rcnn fpn you will see mistakes and many problems...but the score is better with faster-rcnn-fpn!!! I don't know...maybe I'm not good at this 😉
if u have already done with pre-processing plz reply......