I've made good progress, both conceptually and in implementation, and get MAE below 0.1 nowadays. The training sample is small, so I use all data and use the test as a type of hold out and submit to get a type of validation score. But no matter what MAE I get in training sample, when I submit I get something that is worse than all zeros. I think, or maybe hope, the test data that report MAE back to us is dirty or not representative or small in comparison to full test data that will eventually be used.

Yes, that is true, locally you can get pretty good results, but not with the leaderboard public test set.

I've been running into this a while now. I either should split the limited data I have and make my own validation set or play in the dark and submit the model I think will work. Some info on how big the actual test sample is and on how big the one being used to construct the leaderboard will be helpful.

Anyhow, what would your thoughts be on using your own validation sample and ignoring the leaderboard for this one?

I've submitted a few models and kept track of the score both locally and on Zindi - the correlation is -40%.