Congrats to all winners and participants!

This is one of the most challenging problems I've worked on in NLP.

You could guess from my numerous submissions how many ideas I had experimented with. Some failed surprisingly whilst a few did well on private LB, surprisingly lol.

The texts are variably long and , contains mixture of languages (I detected English and Chichewa), and a serious imbalance in classes.

I applied sequence of augmentations(multilingual bert, chichewa fasttext embeddings word substitution, and random word substitution) on all samples belonging to chichewa. To avoid data leakage I kept all augmented samples in train folds(5 stratified folds)

Model 1 - I trained a random forest classifier on tfidf features using the 5-fold augmented chichewa data. Model 2 - I trained Longformer( a transformer model specially designed for long sequences) on all English data using 5-fold stratified splits(no augmentations). Auxillary features such as binned sequence lengths and word-based statistical features were used to train the longformer model. Obtained final submission by merging the English and chichewa submissions.

Text cleaning and FE :- removing - punctuations, integers and english stopwords, and then text normalization by lemmatization(this was a better alternative to stemming)

What did not work(for me):- Pretraining and finetuning( tried bert, multilingual bert and xlm-roberta-base), Chunking each sample into 6 chunks of size 229, concatenating tfidf and ngrams/statistical features, removing chichewa stop words...(some of the major failed ideas)

Wish I tried:- Train the longformer model on all data.

My transformer models did poorly on public LB but great on private. A main reason why I failed to select any solution using transformers just before competition closed.