Arm UNICEF Disaster Vulnerability Challenge
'/%3e%3c/g%3e%3cuse xlink:href='%23b' transform='rotate(10)'/%3e%3c/g%3e%3cuse xlink:href='%23c' transform='rotate(20)'/%3e%3c/g%3e%3cuse xlink:href='%23d' transform='rotate(40)'/%3e%3c/g%3e%3cuse xlink:href='%23e' transform='rotate(-80)'/%3e%3c/g%3e%3cpath d='M-225-21.5h450v7.5h-450z'/%3e%3cpath fill='%23339E35' d='M-225 86h450v100h-450z'/%3e%3c/svg%3e)
Helping Malawi
$10 000 USD
Completed (over 1 year ago)
1193 joined
347 active
Start
Mar 15, 24
Close
Jun 23, 24
Reveal
Jun 23, 24
High MAE values in CV
Help · 18 Jun 2024, 03:48 · 9
Hi everyone,
I am just wondering if it is a mistake from my side or something else, but after splitting data to same split as train/test and calculating the MAE after formulating the predictions similar to the submission (3 rows per id, one for each class), we are getting really high MAE in CV (~4.XX) while good results on LB (~0.28).
Based on the settings we have currently + some intuition, I am sure we are not overfitting, but these results are just weird. Was wondering if other experienced similar values?
For me
CV: 0.279 LB: 0.271
is this your singlefold or 5 fold cv
5folds absolutely
Your cv vs lb score is really good , please do share your approach after the competition ends , I'm pretty sure every participant would learn a lot from it.
@Ecommer
Could I know your cv and lb baseline scores? The MAE scores you started improving from, not the current one.
When evaluating your cv scores, do you make sure the ground truth values for non_existent roof types are factored in the mae scores. i.e There might be no id_xxx_1 = 0 (usually the case for _1) in the ground truth values but if your model correctly predicted id_xxx_1 to be 0, do you let it count in your mae calculation?
Thanks. Yeah they are considered in the predictions and ground truth. So I don't think it is the problem. btw, I am getting quite good scores inside the yolo while training for the other metrics (precision, recall, MAP, ...) So I think it is a bug from my side. Anyway, after some thinking I think f1 score should be correlated to MAE right? I think high scores in either of these metrics should optimize the other in our case here. So I am thinking about focusing on the metrics while training for now.
Sounds like a good idea. From my side, I am optimising for fitness in my yolo model. but I still have over 0.9 locally vs 0.35 lb when it comes to mae
Thanks everyone. Finally solved the bug!