When I try training the model, I get the below error;
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [92,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [93,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [94,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [95,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
0%| | 0/68 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/opt/conda/bin/yolo", line 8, in <module>
sys.exit(entrypoint())
File "/opt/conda/lib/python3.10/site-packages/ultralytics/cfg/_init_.py", line 591, in entrypoint
getattr(model, mode)(**overrides) # default args from model
File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/model.py", line 674, in train
self.trainer.train()
File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py", line 199, in train
self._do_train(world_size)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py", line 376, in _do_train
self.loss, self.loss_items = self.model(batch)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 88, in forward
return self.loss(x, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 267, in loss
return self.criterion(preds, batch)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/utils/loss.py", line 222, in _call_
Thank you. This saved me a lot of time. Fun to play around with YOLO!
Hi @mark98, I'm having issues converting the bounding boxes to yolo format. Your formula on how to convert is this;
x0, y0, w , h = box[0], box[1], box[2], box[3]
yolo_bbox = ((2*x0 + w)/(2*iw)), ((2*y0 + h)/(2*ih)), w/iw, h/ih
Let's say in this case that;
box = [122.0, 1.0, 42.0, 30.0], iw = 500, ih = 500
yolo_bbox = ((2*122.0 + 42)/(2*500)), ((2*1.0+30)/(2*500)), 42.0/500, 30.0/500
Doing the calculations gives yolo bounding boxes as below;
yolo_bbox = 0.286, 0.032, 0.084, 0.06
Is that right.
My issue with this is that yolo bounding boxes are from 0 to 1.0. But this calculations are creating bounding boxes less than the range like 0.032
I need your help on this. Thanks
Hi, I didn't understand your issue, 0.032 is between 0 and 1
When I try training the model, I get the below error;
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [92,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [93,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [94,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
/usr/local/src/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [6569,0,0], thread: [95,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
0%| | 0/68 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/opt/conda/bin/yolo", line 8, in <module>
sys.exit(entrypoint())
File "/opt/conda/lib/python3.10/site-packages/ultralytics/cfg/_init_.py", line 591, in entrypoint
getattr(model, mode)(**overrides) # default args from model
File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/model.py", line 674, in train
self.trainer.train()
File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py", line 199, in train
self._do_train(world_size)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/engine/trainer.py", line 376, in _do_train
self.loss, self.loss_items = self.model(batch)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 88, in forward
return self.loss(x, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/nn/tasks.py", line 267, in loss
return self.criterion(preds, batch)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/utils/loss.py", line 222, in _call_
_, target_bboxes, target_scores, fg_mask, _ = self.assigner(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/utils/tal.py", line 72, in forward
mask_pos, align_metric, overlaps = self.get_pos_mask(
File "/opt/conda/lib/python3.10/site-packages/ultralytics/utils/tal.py", line 94, in get_pos_mask
align_metric, overlaps = self.get_box_metrics(pd_scores, pd_bboxes, gt_labels, gt_bboxes, mask_in_gts * mask_gt)
File "/opt/conda/lib/python3.10/site-packages/ultralytics/utils/tal.py", line 113, in get_box_metrics
bbox_scores[mask_gt] = pd_scores[ind[0], :, ind[1]][mask_gt] # b, max_num_obj, h*w
RuntimeError: CUDA error: device-side assert triggered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
----------------------
I'm positive this error is probably caused coz of the bounding boxes not being in the right format.
I have previoulsy trained several yolov8 models, and I never got such an error.
Could it be instead a label class error? How many class do you set in YOLO model?
I set number of classes as 3 in the yaml configuration for the yolo model.
So nc = 3. Isn't that right
You have to set 4 classes, one for background
Setting to 4 classes brings up this error below;
SyntaxError: /kaggle/working/unicef_dataset/data.yaml 'names' length 3 and 'nc: 4' must match.
I'm thinking the issue I'm having is around the normalization of the bounding boxes
Can you post your code here ?
Kindly share your email so that I can send you the notebook.