Hey folks, I put together a summary of my approach to this challenge.

🗒️ Overall Solution Description

I achieved my best LB score with a baseline model that returns a tweaked version of the input grid. This approach was prioritized after several rounds of EDA and a few hours/days playing around with the very addictive ARC-AGI web app. Fun facts, this wasn't expected as I could see that the LLM-based approach with a better local pixel/cell color accuracy, but it unfortunately received a lower score after submission 🤔

In fact, the Qwen2.5-based pipeline 😎 (Qwen/Qwen2.5-14B-Instruct) can achieve over 32% on the public LB. I noticed a bug in the output generation workflow after the competition ended. I'm now very curious to see if this model can benefit from a few additional tricks.

💻 Running on Mac

For some reason, the Mac version with mlx-community/Qwen2.5-14B-Instruct-8bit (4B params) could not process medium-sized inputs (15x15). The short period for experiments did not allow for more investigation, but I plan to update the solution with the mlx pipeline once fixed.

🖥️ Compute requirements

This solution was developed on both mac- and cuda-enabled host machines. The setup comprised :

• A local dev setup MBP M4 - 24GB
• A cloud-based setup with an NVIDIA RTX 6000 - 48GB

From my experiments, Qwen2.5-14B requires ~15GB on Mac (8-bit); ~29GB (~22GB using 8-bit and ~16GB using 4-bit) on RTX 6000. Quantizing to 8-bit or 4-bit allows for keeping the model within the 24GB limit, making it possible to run it e.g. on an RTX 4090. The unfortunate conclusion is that this approach (especially the LLM-based one) is not GPU poor-friendly 😢. I have always been concerned about compute availability to lower the entry barriers for AI/Data science competition on the continent...but this is probably a discussion for another day 🫠.

It is perhaps importnat to note that all experiments were conducted within a budget of less than $30, which could have been zero (to some extent) if the local dev setup did not crash on some of the grids.

Anyway, you can find my solution at https://github.com/dric2018/zindi-arc-agi (star if you find it useful ;-) It also describes the entire process from day 1.

A compact summary is provided at https://github.com/dric2018/zindi-arc-agi/blob/main/solution.pdf

Feel free to leave a comment in case you want to explore this solution further.

Cheers ✌🏾