Primary competition visual

Arm UNICEF Disaster Vulnerability Challenge

Helping Malawi
$10 000 USD
Completed (over 1 year ago)
1193 joined
347 active
Starti
Mar 15, 24
Closei
Jun 23, 24
Reveali
Jun 23, 24
User avatar
ahmedo42
doubts about evaluation metric
Data · 1 Jun 2024, 10:21 · 6

the current evaluation metric doesn't care about the type of house detected in the image, only about the count of houses in the image, how does this help in differentiating between thatch and tin houses? , shouldn't the evaluation be based on getting the number of each category correctly in the image?

Discussion 6 answers
User avatar
Koleshjr
Multimedia university of kenya

it does actually , since it's not total counts but total counts per roof type

so you have to get the roof type correctly to count for each type

1 Jun 2024, 10:51
Upvotes 1
User avatar
ahmedo42

just to confirm

if we have 3 houses, 1 of each type, the prediction should be 1+2+3 = 6.

User avatar
Koleshjr
Multimedia university of kenya

no , if you look closely at the sample submission for each image you have to identify the type so if an image is called imageone you will have imageone_{rooftype}then the count of that roof type

User avatar
ahmedo42

you're correct, didn't notice that the id is changed in the sample submission

What if you misassign roof type counts? Say for tin houses, you assay the counts for thatch houses and vice versa. Since the metric is MAE, does it affect the score?

User avatar
Koleshjr
Multimedia university of kenya

yes it does