AI4D Malawi News Classification Challenge
'/%3e%3c/g%3e%3cuse xlink:href='%23b' transform='rotate(10)'/%3e%3c/g%3e%3cuse xlink:href='%23c' transform='rotate(20)'/%3e%3c/g%3e%3cuse xlink:href='%23d' transform='rotate(40)'/%3e%3c/g%3e%3cuse xlink:href='%23e' transform='rotate(-80)'/%3e%3c/g%3e%3cpath d='M-225-21.5h450v7.5h-450z'/%3e%3cpath fill='%23339E35' d='M-225 86h450v100h-450z'/%3e%3c/svg%3e)
To achieve both fairness and performance, hosts should limit resources (e.g. single GPU, training time, inference time), not methods (e.g. translation, external data).
In the Social Media Sentiment Analysis for Tunisian Arabizi competition, there is a limit on resources (training time is 8h).
This was written by @ktr. I agree with him - there should be a resource cap in the competition.
Why should this be there?
As we all know, there already exists a pre-trained model `mT5`. I haven't checked as to what it's accuracy would be but that is because I don't have the compute power to fine-tune the whole model (the largest of which is `11 Billion Parameters`). This provides an unfair advantage to competitors with large GPU clusters to fine-tune that model and get pretty high accuracy without doing much work.
Technically, semi-supervised training also poses this exact same risk, that is why I was pleased when it was banned. but this has limited many solutions to be used.
A clean solution
I recommend that @zindi should rather somehow increase the dataset so that it can be actually used with DECENT accuracy enough to be used in real-life by the sponsor. Right now, the focus should be on resource cap to prevent good Data scientists from backing out of the competition.
- Resource cap on runtime, GPU of the notebook and thoroughly checking for any external data (like checkpoints) in the notebook.
- Banning of the pre-trained model mT5. BETTER --> ban the bigger model (above 1Billion Parameters) allowing some competitors to use their knowledge and squeeze out the best performance while it is run under the resource cap.
- A final update to rules outlining the above points, and a promise not to create any more restrictions other than the ones already established.
I would recommend that there should also be a cap on the GPU used --> until `V100` only. That is because Nvidia's `A100` (which is publicly available) has enough VRAM to be used to fine-tune some of the bigger models.
I think participants are being more concerned with winning the competition than with growing their knowledge and expanding the African AI landscape.
Limiting resources seems like a stretch. As an individual participant, one could also argue against the advantage teams have over individuals. More brains, more CPUs (or GPUs), but should teams be banned just to "level" the playing field? No.
The reality is that there will always be resources some will have that are better than others. Be it owning a computer, number of GPUs, level of education/experience, mentorship, number of team members, reliable Internet or consistent electricity, the list is endless. Obsessing over these will make you lose your focus.
There's also no guarantee that using a GPU trained transformer model will outperform other available techniques. Supposing that it is the best method, for the task at hand, doesn't the sponsor also want/deserve the best model for their investment?
So hypothetically speaking, if someone has done less work but uses more resources, he/she deserves a prize than the one who actually put in a ton of time refining, researching, implementing SOTA models from research papers, whose model probably is faster and uses much fewer resources? There is a high relation between the no. of GPU's and CV score which is why Kaggle has a runtime cap. I don't expect Zindi to provide a GPU, but it could at least enforce some rules to help competitors with less GPU requirement.
Competitions should always try to maximize the playing field and make it fair for others. A team would always be better than an individual, but that is something that can't be helped. What *can* be, is to put a resource cap.
If the sponsor truly wanted someone to put in so less work, they could have freelanced it which would be 10x cheaper. But they did not. why? because that's what they wanted - the leanest and fastest model that actually implements some new things to improve accuracy and reduce the number of resources needed to run it. Its supposed to be an effort to push AI forward.
Clearly, it seems like some people want to do their best to disadvantage people with lower resources. Again, I will re-iterate that I don't mind pre-training models - only that their size be limited to a value that can be trained on the free GPU's available on Colab/Kaggle.
You are looking at GPUs as a silver bullet to machine learning and data science. I'm certain a person with a GPU(s) still has to research and refine their models. GPUs do not do that for you, they only accelerate training, validation and testing. Using a GPU will not improve your models accuracy!
The sponsor may want the fastest, the leanest or the most accurate (based on their metric). Nonetheless, it is for the sponsor to decide, not the competitor. Models in this competition are being scored on their performance, not their size or speed. That renders the size argument invalid. And, what if you can't get acceptable performance from "small" models, should the competition just accept the low scores or be abandoned?
FYI: I'm not advocating for GPUs per se. I'm against resource limitations, of any kind, as they are unnecessarily constraining. Everyone has some sort of disadvantage compared to another competitor. My model submission is CPU based and I don't mind.
A team of 4 can simultaneously do 4x what an individual does, regardless of any limitations in place. That gives teams an upper hand *ALL* the time. But, should they be banned? Again, no!!
Form a team or find a team member with a GPU, if you want to beat any handicap you believe you might have.
@tmbenhura
Sorry to point out, but that is wrong. GPU may accelerate training but their real power comes from:-
Distributed computation and low latency VRAM. But most importantly, from competitors point of view, More powerful GPU's allow for extensive hyperparameter tuning and allows new technologies (like Mixed precision training - available only on Colab Pro) for even faster pipelines leading to competitors taking advantage and avoiding the runtime cap.
About team, I have already told - its a factor out of our hands. BUT - some factors are in our hands.
Those factors are compute resources ONLY. Nothing more. Runtime Cap + Pretrained model size cap. That's it.
Most Data scientists know how to optimize and fine-tune their pipelines. Resources are not constrained - its you struggling to make the best of them and disadvantage others. If someone can use Colab to make a model, so can anyone else.
Using GPU's does increase model accuracy if used with the right customizations. but this is something that I do not think I am credible and would rather have @serigne share his opinions on his topic, as a Kaggle Master.
A GPU is not distributed computation, but it can offer parallelised computation, which is different. At the end of the day it all boils to the acceleration of computation. This in turn accelerates your training / validation / testing.
And, any factor can be controlled if you want to, it doesn't mean it should. You could even mandate CPU training only, max core i7. That would make it "fair", because that's the resource I have. But, the competition is NOT centred around me.
Colab requires reliable Internet. Does everyone have that?
Anyway, if a seeded model is trained for the same number of epochs on a CPU/GPU the accuracy remains the same. The CPU model will take longer to train though.
So, like a said, work around your perceived handicaps. You should have a model training in the background, right now, to catch up to the guy with the super GPU. Assuming their model is accurate / useful to begin with. 😉
Not everyone has an i7 here. My laptop has 2GB RAM. But, the competition is NOT centred around YOU. its meant for everyone.
Also, are you really that daft? CPU is nearly 32 times slower than a GPU. Are you kidding me? It would take 64 hours to do 2 epochs compared to what could have been done in GPU in 2 hours!. BUT - This means that a person with CPU would have to wait 64 hours to run a model with different parameters. This reduces your ability to change and experiment with models. This makes you handicapped to not try different architecture even though your knowledge may be good.
Just tell me this - if you are pretty confident in your abilities that you might be better than someone - why are you so afraid to make it an even playing field in terms of resources? Why are you so afraid to give someone else the exact same GPU with exact same CPU and train your model against them using only your skill&knowledge????
That's what separates the Kaggle Masters from others - to improvise in low resources. That's why they get to be at the top and earn more $$. Its also the reason why Kaggle sets a resource cap - to make the competetion fair
I meant to say that the competition in NOT centred around me (and have corrected the post). This is a sentiment you should have gathered given my position with this matter. And, that is why I am NOT bothered about resources. Those that have them, should be free to use them. Those that have an i9 should be free to use it. Those that have 64 GB RAM should be free to use it. Those that have access to Colab should be free to use it. If you have a GPU farm, good for you, use it.
Clearly, I'm not the one who is afraid of others. I am confident will my abilities and lower resources, that I will do well and grow. I may not win, but I will definately come out better on the other side. I have come to appreciate that life is neither balanced nor fair, and ones efforts should be to overcome those obstacles. If they don't ban large models or limit GPU usage, are you going to quit? I won't, I'm not a mercenary just here for the $$ as you put it (not that it wouldn't be great to win).
Well, if Kaggle rules are rules that you want, then maybe that's where you should be. In my opinion, Zindi's policies should not be governed by what Kaggle is doing. If Kaggle decides to allow unlimited GPUs and large models should every other platform follow suit? No, because platforms are independently run, sponsored and administrated. Kaggles reasons for their GPU limits actually relate to their limited budget for providing competitors free GPU access, on their platform, and not to any fairness issues.
Anyway, as your temper seems to be flaring, this is the last I shall comment on this. You do you. Good luck and god speed!! See you on the leaderboard