I noticed that the labels are inconsistent with each other. Take these two as an example in hurricane_florence_2018.jsonl

{"tweet_id": "1041626829407354881", "user_id": "2458176704", "text": "Three days after #Florence made landfall, the storm is still battering the Carolinas. More than 30 American Red Cross volunteers from Connecticut are stationed in the south to help with", "created_at": "Mon Sep 17 09:56:03 +0000 2018", "humAID_class": "rescue_volunteering_or_donation_effort", "location_mentions": [{"text": "Connecticut", "type": "State", "start_offset": 134, "end_offset": 145}]}

{"tweet_id": "1039438451031113728", "user_id": "3108351", "text": "Authorities order mandatory evacuations in the Carolinas and Virginia as rapidly strengthening Hurricane Florence approaches", "created_at": "Tue Sep 11 09:00:13 +0000 2018", "humAID_class": "displaced_people_and_evacuations", "location_mentions": [{"text": "Carolinas", "type": "State", "start_offset": 47, "end_offset": 56}, {"text": "Virginia", "type": "State", "start_offset": 61, "end_offset": 69}]}

Notice that both have 'Carolinas'. However, in the first tweet, Carolinas is considered a location mention while in the second tweet it isn't considered. Wouldn't this be a problem when evaluating the model? Perhaps some postprocessing should have been done on the labels which would have caught such issues.

Another example using line 2 and 3, same dataset. One mentions Florence as a town but it doesn't mention the same for the other tweet. However, Florence is the name of the earthquake.

{"tweet_id": "1040157749353566208", "user_id": "1014614947907756033", "text": "Florence, now a Category 2 hurricane, is forecast to unleash extreme storm surge, possibly historic flooding rains, and damaging winds in the", "created_at": "Thu Sep 13 08:38:27 +0000 2018", "humAID_class": "caution_and_advice", "location_mentions": [{"text": "Florence", "type": "City/town", "start_offset": 0, "end_offset": 8}]}

{"tweet_id": "1041228731099086848", "user_id": "37509828", "text": "RT @annamoor1995: Rescue the #Pigs #Trapped During Hurricane Florence @moveon", "created_at": "Sun Sep 16 07:34:09 +0000 2018", "humAID_class": "rescue_volunteering_or_donation_effort", "location_mentions": []}