the current evaluation metric doesn't care about the type of house detected in the image, only about the count of houses in the image, how does this help in differentiating between thatch and tin houses? , shouldn't the evaluation be based on getting the number of each category correctly in the image?
it does actually , since it's not total counts but total counts per roof type
so you have to get the roof type correctly to count for each type
just to confirm
if we have 3 houses, 1 of each type, the prediction should be 1+2+3 = 6.
no , if you look closely at the sample submission for each image you have to identify the type so if an image is called imageone you will have imageone_{rooftype}then the count of that roof type
you're correct, didn't notice that the id is changed in the sample submission
What if you misassign roof type counts? Say for tin houses, you assay the counts for thatch houses and vice versa. Since the metric is MAE, does it affect the score?
yes it does