Primary competition visual

Radiant Earth Spot the Crop Challenge

Helping South Africa
$8 800USD
Challenge completed ~4 years ago
Classification
Earth Observation
548 joined
101 active
Starti
Jul 05, 21
Closei
Oct 03, 21
Reveali
Oct 03, 21
Memory Error in Google Colab
Notebooks Ā· 16 Aug 2021, 21:01 Ā· 3

** This question is duplicated in the 2 forums because there are different people in each one. Sorry for that

Hi, I´m getting an Memory Error when I try run the base notebook in Google Colab: https://github.com/radiantearth/mlhub-tutorials/blob/main/notebooks/South%20Africa%20Crop%20Types%20Competition/Radiant_Earth_Spot_the_Crop_Baseline_Model-S2.ipynb

The same error occurs in https://github.com/radiantearth/mlhub-tutorials/blob/main/notebooks/South%20Africa%20Crop%20Types%20Competition/Radiant_Earth_Spot_the_Crop_Baseline_Model.ipynb

During the initial step of processing all the 'tile_ids_train', at the end (90%), the session stopped during the process of the 'tile_ids_train' or after the creation of the X variable, when a pd.dataframe is created: (some time, a little earlier)

I´m using the default configuration of Google Colab, without attending the requirements.txt, but I think that is not the problem.

It´s possible run this notebook in Google Colab or only in Google Colab Pro?

Discussion 3 answers

1. Try adding dtype as numpy.float16 "X = np.empty((0, 6 * n_obs), dtype=np.float16)" for the arrays (numpy.int32 for the labels/field_ids arrays)

2. set , out_shape=(N,N) while reading using rasterio. i.e(N like (128, 128), (96,96)) e.g: b8_src.read(1, out_shape=(96,96))

3. use gc — Garbage Collector in the for loop.

BTW.. how long did your code take to run while extracting the pixels on google colab default settings.

Thanks for your response. I'll try your suggestions (the number 3 I had tried without success). My code ran for about 1 hour to extract the pixels.

You're welcome.. if all fails try to use less data