Nice one bro.

Been stuck on 70% using logistic regression for a long time.

I am open to forming a team with you if you don't mind

Email: bammy2050@gmail.

Thanks for sharing.

Just noticed this,

in couple of places you use, the label encoder

#label encoding columns
columns = train.columns.tolist()[1:11]
test_columns = test.columns.tolist()[1:11]

le = LabelEncoder()
for each in columns:
    train[each] = le.fit_transform(train[each])

for column in test_columns:
  test[column] = le.fit_transform(test[column])

here you fit and transform the train and test set separately which is not a good idea. You want to learn label encodings only from the train data and apply on the new test data (by using "transform" operation) - there are a few categorical fields that have different levels in the train and test data and you will need to have code to handle new levels in the test data (one such way is regrouping levels occuring < 1% into a new category). An additional point with using label encoder is some features have several hundred levels so you also add sparsity.

same with the approach of combining train and test data, to generate dummies or other feature engineering (in real life you will never have access to such data to do this step) - this introduces leakage as the model will learn from data it's not supposed to see. Unfortunately, it's quite a common practice in kaggle and other competitions to use this approach (of combining train and test data but it's not one you will use in real life).

len_train = len(train)
new_df = pd.concat([train, test], sort=False)

#getting categorical dummies
categorical_columns = ["ProviderId", "ProductCategory", "ProductId", "ChannelId"]
new_df = pd.get_dummies(new_df, columns=categorical_columns)
new_df.head()