Currently the evaluation metric for this competition is RMSE. One important characteristic of RMSE is that it is sensitive to outliers, increasing exponentially the further a prediction is off. This can be problematic when working with an imbalanced dataset as a small number of large errors will significantly increase the overall RMSE.

The dataset for this competition is VERY imbalanced. Looking at the training dataset, most of the rows have a Yield somewhere between 100 and 1000, however there are roughly a dozen rows way outside this distribution with extremely high Yield going as far as 16800 (I will assume that this is the case in the test dataset as well). In my experience so far these outliers are difficult to predict, but getting them wrong will massively impact the overall RMSE.

Example: after some very basic cleaning and feature engineering, my model currently achieves a RMSE of 435 in 10-fold cross-validation on the training dataset. When I delete the top 20 rows with the highest Yield from the training dataset, this same model improved to a RMSE of 156(!!) in 10-fold CV. So by only removing about 0.5% of the training dataset, the performance of the same model almost triples when measuring RMSE. If I only drop the top 4 rows with the highest Yield I get a RMSE of 275, still an enormous improvement over my initial RMSE of 435 by removing just 0.1% of the training data.

My point here is that the distribution of the dependent variable of this dataset is massively skewed, causing a very small number of the entries to have a disproportionally large impact on the overall RMSE of a model. A much more suitable evaluation metric for a dataset like this would be something like MAPE, which will still punish your score when you are way off predicting this small number of outliers but it does not increase exponentially with the error.