Hello,

In this competition, I learnt the importance of not just considering the CV score but also CV evaluation process in doing a comparison between models evaluated with two different processes.

My team mate and I had two different models developed using two different approaches, one's evaluation showed CV 635, other model evaluation showed CV 700.

Looking at CV scores alone, seems model of 635 is significantly better. Public LB was great for CV 635, exactly same score. However, private LB performance is LB 720 for CV 635 and LB 676 for CV 700. So huge LB shakeup. When we teamed up, we relied on two variants of the CV 635 for private LB tag cuz it seemed better.

Looking back, the CV process being used for both models was different, hence one was evaluated more favourably than the other. Matching the CV process for the two models indeed shows untagged model is better than tagged model.

I think it's important to match CV evaluation processes when we are dealing with model evaluation in any scenario, otherwise our evaluation might already have induced bias and lead to less generalisable solutions.