@AZZASSIN Infact i was totally agreed with all what you've said. This competition is not a good Example for Submission Selection. I was just laughing myself when i discovered that i had a very best score submitted like 25 days ago. Just a single model (LGBM) with 0.4984 Public Score and with 0.5040 Private LB Score. The funniest thing is that I keep on ensemble my models scores and it keeps increasing my LB score. Lol......

Lol.... What a challange!!! But nevertheless, the best approach is to practice what you've learnt in other "Good classifications competitions" like you said.

Well surprisingly, for me, complex models never performed well even on the private LB. My best model was just a Naive Bayes model. I just discovered that an ensemble of Naive Bayes variants had the best (~0.503), although I didn't select it.

Thank you very much @ASSAZZIN for sharing your experience and your solutions. I have been a victim of overfitting the public leaderboard.

I had a submission which had a cv score of 0.501, a public leaderboard score of 0.49 and a private score of 0.505. I don't understand why the scores were varying so much. Maybe train and test datasets had different distributions or the public leaderboard and the private leaderboard data had different distributions. I am not sure, I did not do proper EDA to check whether this was the scenario. I had given up after seeing that my scores were not improving on the public leaderboard.

Next time, I will not give up as early as I did.

I have been using catboost in many competitions because it usually performs well without tuning hyperparameters.

I agree that there are other competitions which the public leaderboard and cv scores reflect the scores on the private leaderboard. In this competition, it was difficult to identify which submission would perform best on the private leaderboard.

There are times which a competitor can choose to completely ignore the public leaderboard.

This sounds interesting @Gozie. How did you go about ensembling?

By averaging the predictions of the variant Naive Bayes models. The variant models (from scikit-learn) were the multinominal, categorical, and complement Naive Bayes models.

Well Yes it happens a times especially with this kind of competition. Naive Bayes!!! Awesome!!! 👍

Yes you're right. I think It's more about the kind of datasets provided. If a created EDA Notebook portrays or shows a lot of weird things, then It will be very difficult to know if your model performs better or not. So it is more of dataset provided not so much about the model implemented.

That is brilliant @Gozie. I had never thought that Naive Bayes would perform well in a competition.

I agree with you @MICADEE.

Thank you @SanJose. You have shared a good way of selecting submissions.

I did the challenge for an hour or two, then I gave up after realizing that my scores were not improving.