AutoInland Vehicle Insurance Claim Challenge
'/%3e%3cpath id='b' d='M658.84 441.31c-7.618 16.446-22.845 19.271-36.164 5.995 0 0 5.354-3.783 11.086-4.826-1.075-4.574 1.13-10.902 4.235-14.324 3.258 2.227 7.804 6.689 8.96 11.874 8.03-1.04 11.883 1.282 11.883 1.282z'/%3e%3cuse xlink:href='%23b' transform='rotate(9.37 700 804)'/%3e%3cuse xlink:href='%23b' transform='rotate(18.74 700 804)'/%3e%3cpath fill='none' stroke='%23f8c300' stroke-width='16' d='M603 478a340 340 0 0 1 194 0'/%3e%3cg transform='translate(700 380)'%3e%3cg transform='translate(0 -140)'%3e%3cpath id='c' d='m0-513674 301930 929245-790463-574305h977066l-790463 574305z' transform='scale(.00005)'/%3e%3c/g%3e%3cg id='d'%3e%3cuse xlink:href='%23c' transform='translate(-70 -121.244)'/%3e%3cuse xlink:href='%23c' transform='translate(-121.244 -70)'/%3e%3cuse xlink:href='%23c' transform='translate(-140)'/%3e%3c/g%3e%3cuse xlink:href='%23d' transform='scale(-1 1)'/%3e%3c/g%3e%3c/g%3e%3c/svg%3e){kind=small}
If you are doing EDA, you probably noticed that many duplicated rows still exist. How:
ID is not duplicated. Only features are duplicated. if you run the below code you will see it.
data[(data['Policy Start Date'] == '2010-02-11') & (data['Policy End Date'] == '2011-02-10') & (data['ProductName'] == 'CVTP') & (data['Age'] == 25)]
Car_Category should be truck for all
if you look at iveco with red/ red & white - they are duplicated
Interestingly if we take one duplicated row, one of them tells us that target is 0 but the other one tells 1.
because of this our algorithm gets confused which one to believe.
Because of this we need to delete these duplicated rows as they do not make sense from both real world and statistical sides.
So @Zindi could you tell us which one to delete the duplicated rows with target 0 or 1.
When we are close to competition , with 2 weeks pending no changes in data should be done. Whatever it is, need to contented with that, as its same for everyone.
They can extend the competition if the data still contains duplicated rows though, I mean the data has been difficult to crack since so.
If you're really sure there are duplicated rows, you can try using both targets [0 or 1] seperately to test which would yield better result with your model. I'm sure removing duplicate train values is not against the rules of the competition.