You're definitely not alone haha— the CV/LB gap here is brutal. A few directions that helped us stop feeling like it's pure gambling:
- Random K-fold basically lies to you here — train and test are different time periods, so in-distribution CV looks amazing and means nothing. I tried validation that mimics the test regime and even that barely correlated. In the end I treat the LB as the only honest judge and spend submissions like they're gold (one clean experiment per sub, change one thing at a time).
- Prefer signals that don't drift between periods — relative/ratio-style features tend to survive the shift far better than absolute values.
- Think about what physically separates the target from the things it gets confused with, and try to isolate that one specific signal rather than throwing everything at a big model.
- The two scored columns are independent — worth optimizing each on its own terms instead of as one number.
Honestly it's mostly disciplined probing plus a couple of the right ideas, not magic. You're closer than it feels — keep at it!
Thanks man! Honestly I almost gave up on this. It is tough😅. So far tree models are doing horrible man. I don't know whether it is only me. No matter the kind of features I feed the models, whether normalized features, whether ratio-style features, aggregations, training each month as a training point, this competition is really tough🤣. Congratulations to you guys. Would love to learn after the competition😅.
Ha, don't blame the trees - I'm on gradient boosting too (CatBoost-family, nothing exotic), so the model isn't the bottleneck. If tuning models and swapping features isn't moving the LB, that's a hint the lever is elsewhere.
Two nudges:
- Don't let CV pick things — here a feature can look dead in CV and be gold on the LB (and vice versa). Trust the LB, painful as it is.
- Months aren't independent points — training each month as its own row loses how a pixel behaves across the year, which is where a lot of the signal lives.
You're definitely not alone haha— the CV/LB gap here is brutal. A few directions that helped us stop feeling like it's pure gambling:
- Random K-fold basically lies to you here — train and test are different time periods, so in-distribution CV looks amazing and means nothing. I tried validation that mimics the test regime and even that barely correlated. In the end I treat the LB as the only honest judge and spend submissions like they're gold (one clean experiment per sub, change one thing at a time).
- Prefer signals that don't drift between periods — relative/ratio-style features tend to survive the shift far better than absolute values.
- Think about what physically separates the target from the things it gets confused with, and try to isolate that one specific signal rather than throwing everything at a big model.
- The two scored columns are independent — worth optimizing each on its own terms instead of as one number.
Honestly it's mostly disciplined probing plus a couple of the right ideas, not magic. You're closer than it feels — keep at it!
Thanks man! Honestly I almost gave up on this. It is tough😅. So far tree models are doing horrible man. I don't know whether it is only me. No matter the kind of features I feed the models, whether normalized features, whether ratio-style features, aggregations, training each month as a training point, this competition is really tough🤣. Congratulations to you guys. Would love to learn after the competition😅.
Ha, don't blame the trees - I'm on gradient boosting too (CatBoost-family, nothing exotic), so the model isn't the bottleneck. If tuning models and swapping features isn't moving the LB, that's a hint the lever is elsewhere.
Two nudges:
- Don't let CV pick things — here a feature can look dead in CV and be gold on the LB (and vice versa). Trust the LB, painful as it is.
- Months aren't independent points — training each month as its own row loses how a pixel behaves across the year, which is where a lot of the signal lives.
GG man, you've got this!
🙏🙏