lol ... yeah I like that, won't tell my cat about it though.

While I did not find joy with impute, it is really nice to hear about models built around that. I also tried clusters at some stage and got a bit of a lift, but, since my data had missing values, I had to dance around that so much I eventually discarded the clusters. But I think that would be a real nice and natural way to add features here.

The nice thing with impute, if you get it right, is that it allows you to try out many more models as you also did. It also allows you to do some nice, expligit modelling around region. Region had by far the biggest impact and I tried to exploit that a bit but given my GBM "constraint" I did not really find a way to make it work harder, except to stretch some features around it. The problem is that this introduced co-linearity and I was not sure if it helped or damaged the final result ...

Oh well, nice, thanks for sharing!

" the road to mastery is through the basics. wax on, wax off" - Mr Miyagi

fwiw I used mostly KFold, but, right at the end, used stratified KFold with stratification based on region. I think, given the huge size of this, you could sample (as you did e.g. 100k rows) and still get a very good model, as long as you stratified on region and perhaps also on target.

That said, waiting a few hours for a model to complete is pretty much business as usual. I think my final model took ~10 hours ... no matter, just let it run through the night (or through Sunday lunch e.g. yum).

I have old hardware but lots of RAM, so I can handle this type of thing. But I also recently got myself a fancy GPU system. This was one of the first comps I did on my fancy new GPU system. I think the GPU card alone is more than 10k. But, if I get the 10k, it is definitely going into the piggy bank for the next GPU.

Come on Sasol, we worked so very very hard on this, 10k is too small. You can sponsor some of these competitors with souped up GPU enabled machines please. Really ... Sasol, when you evaluate the model, if you think you are getting value, why not reward with a proper GPU machine (price ~50k and indispensable for these comps nowadays). Even the bokke have MTN and FNB behind them ... we really need you here.

... and, I hasten to add ... wow! You did really well, especially for a first comp. Scary, this comps shows anybody who cares that we have skills a plenty. Wooooow - congrats my friend e-x-c-e-l-l-e-n-t!!!!

Stacking here is also excellent but you have to do it right, then your score should increase going to private from public. Note many top guys have better private than public, I guess many did that through some kind of ensembling.

Thank you. lol, yeah that 10k is going to help.

Yeah this was also my first comp, I had previously done project work here and there but never had to deal with such messy data...

I actually wanted to try to build an ensemble model with LightGBM & XGB model training on unfilled data, XGB model based on the cleaned & filled data and tehn using a voting system to get the best estimate from the 3 however my skill level(or hardware for that matter) isn't at that level yet so I'm still researching and I'll just do it for curiousity sake

But a huge well done @SboneloN

Yeah look at that - your stacking gave you better private than public!

Colab .... nice option on paper but not really ever worked for me, but probably because I use the cheapo option. But afaik they don't yet do SA, otherwise I would have bought an account, but now I have my fancy GPU so no worry. And, nice big backup system to keep the lights on when Eskom fails us. My gpu adds a considerable load to the system fwiw, but at least in summer there is enough sun, might struggle in winter.

You can share notebook but I'd suggest just wait until Zindi finalise and seal the LB.

How much RAM in your machine?

I have to get a backup too. In terms of the GPU, I only have an MX450. What are you using? PC RAM is 16 GB. I also use the free version of Colab thus the difficulties.

Thanks for the advice. I will wait till the LB is finalized.

Ok, 16G is a bit small for this type of a thing. My very old (2017) machine has 32 that I added myself and at some stage I could do any comp with that. My fancy new GPU has 128. I ordered 256 but when I saw the price I had to scale back a bit ... the GPU itself is 4070 if I remember correctly, but it has only 12G GPU ram and I've had quite a few keras models that would not work in that. So I am saving for the next one and trying to convince my wife it is a good idea to invest in GPU RAM ...

Its a games machine - the rep told me it is best value for money, which I think is true, but it comes with fancy spinning lights. But the bling grows on you after a while.

GPU here made a big difference for me. I tried on CPU only and did some prelim stuff like that, but the difference is so big as to be impractical to use CPU for this one.

I will have to upgrade to 32 GB seeing I have the competitive coding bug now. Good luck in getting the extra funds to invest in the GPU RAM :)

The games' machines seem to be good at this. Will look into them as well.

?

Did you really do all that? Wow @wuuthraad impressive, as always. Looking forward to the rest. This is thing with you, always feels as if you are two steps ahead with the tech and has this guilty-I-must-do-that-also kind of feeling to it, you know, why throw resources at it if you can do it smarter.

Yeah, probably should read that article and learn those things you mention ... thanks, now I have real motivation to do it. Any my wife will support it, as it will stave off the imminent GPU upgrade a bit ...

Can you see it now @skaak?

@skaak and @Rorisang simplify you life guys, use Rapids. Hahaha.I am touched you two were training your models and waiting for over an HOUR!. Yes there are exceptions but this is not one of them

I was just FEing to the moon.

Also... one of the steps I forgot to mention was , I used "undersampling" when training my model to get ~ more even distribution of the target variable

He he, I thought you edited it out a bit while the LB is still open. Nope, just see a dot.

Wait wait wait, there it is now.

Wow, you are not sparing the details ... I have to thank you kind sir that is nice.

I've used h2o in the past but it added too much complexity and I stay away from such overhead, but you know, if it (rapids) is as good as you describe then it is worth it.

"Undersampling" e.g. select same amount of + and - and thus undersample the many -? So it made the sample smaller? Is that not why you had good performance then?

Anyhow, will leave these for the live session, but definitely will check out rapids.

Initially I used lots of CalibratedClassifier, hoping it would improve results, but eventually ditched it for the time saving and it did not really make a difference as far as I could tell.

Woooooow thanks again.

I stayed away from optuna, it is too easy to overfit (I think), but I do a few very small gridsearches on hyperpars. Perhaps one day I will have to use optuna ... I tried RF. I think any tree based method here would work well, but given the time, had to cancel it eventually. If you could do RF with rapids it certainly is very impressive. Wow, need to read! it. RF + GBM I think would nail this one.

I'll share my notebook when @Zindi gives the all clear. knowledge is meant to be shared

@wuuthraad, I heard of Rapids (only knew of rapyds) today from the discussion you had with @skaak. I will investigate it. Thanks for the links.

Grid searches will definitely work but take a long time. What do you think of random search instead?

I oversampled instead so as to have 'more' data but the approaches curb the same problem.

Random search - depends but sometimes can be the right thing to do, and, given the complexity of reality, any model is a type of random search (on a philosophical level ...)

The defaults are often good, and set by experts. To deviate you need some good reason, but it is always good to test with a simple grid around the defaults if you have good reason, and see the impact, or to deviate from the defaults to more heavily regularise the model.

The nice thing here is that there was lots of data .. sure a modern problem will easily have even more data, but you did not have trouble sampling or any sort of degrees of freedom problem.

On the issue of oversampling and undersampling... I did both as the training and testing (on the entire dataset) gave me a more generalized model so if I got an F1 scoore in training then I'm almost certain that's the score I will get in the LB to save on having to use submissions and time trying to figure out if the changes made were impactful.

For the oversampling I used a threshold of 0.7, 0.6 for undersampling and passed it through pipelines to get a more mixed dataset

The downside is that I could not build an accurate model due to class imbalance(60% +; 40% -) but LightGBM can handle imbalances well. However I was able to get the F1 score to 0.701 however it was only 0.689 in LB due to low precision but a huge improvement form the 0.680 I was at

I'm really thankful for the tip on rapids... I'm out here using a 6GB RAM GPU with 12GB RAM disk looking at @skaak's conversation with complete envy

Same here. I can only envy @skaak 2. I was doing all the standard preprocessing steps so that I won't have to question the methodology. So I did the over/undersampling just to cover my bases.

As my kid's piano teacher used to say: Don't blame the equipment.

I've desperately needed a GPU for years, only recently got it, and now don't know what I did without it, in fact I want two more ... .

But without GPU, you can still build a decent model. This one in particular, just "normal" data, no images or DNA or video or audio. And with stuff @wuuthraad mentioned or using e.g. HDF5 you can get around the mem limitation. Models like GBM will anyhow sample the data and not use all of it, especially if the data is quite big.

Then there is always stuff like colab or kaggle ...

Ag wat, I feel your pain! Come on Sasol ... and other hosts and sponsors ... we need equipment desperately.

Wow @MICADEE thanks for sharing. Only now do I realise the non-SAers have been removed, nice ensemble ... as I mentioned earlier, you need this to lift private score. My best private was 0.70059... but I did not selected it. I got (based on current LB) a winning candidate at around 50 (I used *all* 100 subs) with my first 0.699.. sub (privagte = 0.7002...)

Oh well, but let's not swap fish tales ... yes, it would be nice if this was completely open, and of course a lot more difficult.

I struggled quite a bit to get a decent score, and final and first model are not that different, except for features, so I have to wonder how you did it so quickly? How do you generate features for this one? You make it sound easy ... I had to burn subs and time and effort to generate them ... any insights you may want to share?

...fwiw even towards the end I was still experimenting with features, trying to find the one that would make it all click. I think there was not one, but several you had to have ... but please, any insights would be valuable ...

@Skaak... Smiles.... Awesome!!!. On how i did it quickly... It's as a result of devoting my time to Feature Engineering from the beggining without taking modelling into consideration at that moment. I put all manners of what i know into this Feature Engineering part as i knew that it will be very hard to rerun this F.E notebook all over again. Infact, my main target was to break the record of achieving 0.7xxx on Public LB (which is very much possible) but unfortunately i ran out of memory when i implemented all my ideas on this F.E aspect. Thus, i had to do away with few parts of these F.E ideas and only stayed with the ones that will contain my colaboratory memory. In fact, i wanted to pay for colab pro in the first place but once it's been mentioned that the project was strictly meant for south africans, i had to think twice on this. The following are the steps taking on my F.E aspect among other ideas that were excluded due to memory shortage of free colab:

1. Re-categorizing feature "tenure".

2. Aggregation of few numerical features

e.g agg_columns:'revenue','Amount','arpu_segment','data_volume','on_net' and groupby on cat_cols like: 'region', 'tenure'. Using statistics: {'25%': 'p25', '50%': 'p50', '75%': 'p75'}

3. Feature Intereactions.

4. Label encoding strategy:

from tqdm import tqdm

for col in tqdm(df_train.columns.drop('Target')):

if df_train[col].dtype == 'O':

df_train[col] = df_train[col].fillna('unseen_category')

df_test[col] = df_test[col].fillna('unseen_category')

le = LabelEncoder()

le.fit(list(df_train[col]) + list(df_test[col]))

df_train[col] = le.transform(df_train[col])

df_test[col] = le.transform(df_test[col])

df_train[col] = df_train[col].astype('category')

df_test[col] = df_test[col].astype('category')

else:

df_train[col] = df_train[col].fillna(-99999)

df_test[col] = df_test[col].fillna(-99999)

print('Missing data in train: {:.5f}%'.format(df_train.isnull().sum().sum() / (df_train.shape[0] * df_train.shape[1]) * 100))

print('Missing data in test: {:.5f}%'.format(df_test.isnull().sum().sum() / (df_test.shape[0] * df_test.shape[1]) * 100))

num_cols = X.select_dtypes(include = 'number').columns.to_list()    # numerical features

cat_cols = X.select_dtypes(exclude = 'number').columns.to_list()    # categorical features

NOTE: Possibility of attaining 0.70xxxx on Public LB:

By adding categorical column "top_pack" to the set of categorical columns used in my step 2 above, then one can easily get the job done actually but the memory shortage issue didn't allow this. Though the colab pro is worth paying for had it been other participants other than south africans were allowed. This is just a brief of few things done on this great project.

NOTE 2: I didn't even use GPU in my modelling aspect with both LGBM & CATBoost used. Total time taken for the two models ~ 1h 55mins.

Cheers !!!

Hi - thanks for sharing, I appreciate. Here I think you had to handle top_pack correctly to get good score ... it was one of the things that gave me a bit of a lift after being stuck for a while. I initially tried to simplify top_pack as there are so many possibilities, but you had to try them all (and then throw away based on feature importance the ones that did not contribute, so bit of both).

Oh well - I tried to do the feature interaction right here, with a mathematical approach of course. So I looked at products and ratios of important features and also tried to understand features and based on that combine them in products or ratios or logs etc. I also did some feature stretching, which added just a little bit, but it was significant nonetheless, but I was very weary of it, as it introduces (more) multi-colinearity in an already very correlated feature set. So I tested it rigorously and it still worked, but I think it would definitely contaminate the feature importances a bit.

At some stage I added new features by recording direction in which leaf node is reached. This is funky way to generate new, non-linear features I read about, but it did not help at all, so I returned to my mathematics.

This was lots of fun - in a way it is simple as you also describe. I think if you sort of do default encoding and GBM correctly you should get ~0.67, but then to go from there to 0.7 (I only reached 0.6999 on public) was quite a journey, but, of course, also a nice one.

btw how many features did you have in the end?

@skaak Great !!! 👍. As per number of features generated. Yeah.. 266 features in total.