I've shared my solution for the 6th place on the private leaderboard. Hopefully, it will not change after the code review :)

https://github.com/letfoolsdie/zindi-agricultural

In summary, the final solution is a geometric mean of several imagenet-pretrained models, trained with different parameters on spectrograms/melspectrograms, averaged first by folds and then with each other.

There's also a postprocessing, where I try to find junk test files (containing just noise) using pretrained PANN and replace models' predictions for them with constant prediction based on frequency of each class. It reduced loss a bit (by ~0.01-0.015 points)