Hello! We finished 3rd on the final leaderboard. We've made all our code and methodology for our model – which we have called XYieldBoost – available here.

We used an ensemble method that took the average of three predictions made using tree-based methods, specifically: XGBoost, CatBoost, and LightGBM. All three of these are tree-based methods, which are great for handling tabular data with non-linear relationships. We also used cross validiation to train and test our model in order to reduce overfitting.

However, I think the most important part was the data cleaning stage. We probably spent 95% of the time on data cleaning, which involved working to understand the variables and reading literature on which variables affect crop yield in Bihar/India. As a result, we learned about different agircultural traditions in North Bihar vs. South Bihar (which is more agriculturally productive and employs the Ahar Pyne agricultural system, leveraging channels and retention ponds to manage water resources and adapt to Bihar’s unpredictable weather). We also learned about the importance of the monsoon in Bihar agricultural cycles. Kharif crops, such as rice, are sown during the monsoon season from June to September and are watered by monsoon rainfall. These crops do well with high rain in Winter. Rabi crops, such as wheat, are sown in mid-November – preferably after the monsoon rains are over – and are watered by percolated rainfall. These crops are spoiled by high rain in winter. We also learned about nitrogen cycles, fertilizer application methods, and irrigation techniques.

With all of this information, we had a pretty good idea that the region in which the crops was grown was important (North vs. South Bihar), as were the various dates on which key agricultural steps were taken, as were the fertilisation choices. We engineered our data to reflect this, and selected the top variables using recursive feature elimination with cross validation.

Overall, we only spent around 5% of the time building the actual model (and only started fine-tuning the model two days before the deadline). The trick is to understand the data and not overfit to the public test set :)

We also took additional steps to ensure that the end user of the data (Digital Green) will be able to make as much use as possible from our results. To provide them with additional insights, we calculated Shapley values for each variable, which are the average expected marginal contribution of one variable after all possible combinations have been considered. These can be seen similarly to the variable importance indicators that can be extracted from random forest models.

Digital Green may be able to leverage the information provided by the Shapley values to improve crop yield for Bihar farmers. For example, as seen in the above plot, applying organic fertiliser directly to the soil in the previous crop cycle was very consistently associated with higher yield per acre. It might then be worthwhile for Digital Green to spread this information to farmers, or possibly even invest in directly giving them fertilisation tools that enable direct soil application.

We also developed methodology that will enable Digital Green to determine exactly which factors are driving specific predictions for a given plot of land. The above plot shows which variables influenced the prediction for ID_1YX1Q0KU8H4J, a 0.375 acre farm in the small town of Noorsarai in the Nalanda district of Bihar. This farmer harvested his crop on October 20th, 2022 – earlier than other farmers in the sample. Typically, Kharif crops such as rice are harvested at the end of monsoon season (October–November). However, the 2022 monsoon was slightly drier than normal in Bihar. This might have meant that earlier harvesting would have prevented crops drying out due to the lack of accumulated monsoon rainfall. As can be seen in the above plot, choosing to harvest earlier was associated with an approximiate increased yield of 37 units per acre. However, this farmer also did not apply organic fertiliser to his soil in the previous crop cycle. Since this application method was very consistently associated with higher yield, our model estimated that this decision potentially reduced the farmer's yield by 25 units per acre.