OutOfMemoryError when running on Google Colab (T4 GPU) and Kaggle (GPU-P100)
Notebooks · 15 Oct 2024, 19:06 · 6

When I am running the notebook starter I have the following error:

ON GOOGLE COLAB: OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 MiB. GPU 0 has a total capacity of 14.75 GiB of which 41.06 MiB is free. Process 3682 has 14.70 GiB memory in use. Of the allocated memory 14.51 GiB is allocated by PyTorch, and 43.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

ON KAGGLE: OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 MiB. GPU 0 has a total capacity of 15.89 GiB of which 21.12 MiB is free. Process 2693 has 15.86 GiB memory in use. Of the allocated memory 15.48 GiB is allocated by PyTorch, and 66.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Please, Does anyone know how to explain that error and to workaround this error?

Regards.

Discussion 6 answers
User avatar
combatant101

Reducing the imgsz or the batch size usually does the trick. If not use both GPUs on Kaggle by setting device='0,1'. This is if you can set accelerator to x2

15 Oct 2024, 19:28
Upvotes 1

Please @combatant101, what do you mean by setting the device to '0,1'?

User avatar
combatant101

It is a parameter which works in Yolo particularly and if you have access to more than one GPU.

User avatar
MuhammadQasimShabbeer
Engmatix

Thanks I have Facing the Same problem as well

24 Oct 2024, 15:00
Upvotes 0
User avatar
MuhammadQasimShabbeer
Engmatix

How to USe TPU for Yolo

24 Oct 2024, 15:02
Upvotes 0

Did anyone find a solution to this issue? Need help. I reduce the batch size and the image size, but still the same problem. I deleted the torch caches too.

26 Oct 2024, 01:18
Upvotes 0