You said "I've found that YOLO struggles with detecting small objects effectively".

1. What is your image pixels size for this approach?

2. Did you try larger pixel sizes for this approach? (Like image sizes between 512 * 512 to 1024 * 1024) if yoy try larger pixels your model can detect small objects effectively.

Model suggestions:

https://docs.ultralytics.com/tasks/classify/#models

In above link you can see that ImageNet - 1k yolov8 results. Yolov8 didn't perform well for ImageNet - 1k dataset. It shows yolov8x gives top - 1 accuracy 79.0 with flops 154.8 B.

So, you want to try well performing ImageNet - 1k top - 1 accuracy models, like SE_ResNeXt, EfficientNet, Swin, ViT, etc...

Approach Suggestions:

1. Mask R-CNN, 2. Faster R-CNN, 3. Cascade R-CNN

In this approaches use better Backbones and fine tune the hyper parameters wisely...

If you try this approach and backbone wisely you can get better results than yolov9.