Cuda Out Of Memory Error

[gauraviiita]

Hello,
I am getting this error. Could you suggest a solution for it?

$ python run_realworld/run.py --config configs/drawer_open.yaml
/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
WARNING:root:Failed to import geometry msgs in rigid_transformations.py.
WARNING:root:Failed to import ros dependencies in rigid_transforms.py
WARNING:root:autolab_core not installed as catkin package, RigidTransform ros methods will be unavailable
/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight'])
  -> loaded checkpoint assets/ckpts/minkuresunet_kinect.tar (epoch: 10)
  final text_encoder_type: bert-base-uncased
  Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight']
• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight'])
  Traceback (most recent call last):
  File "run_realworld/run.py", line 132, in
  main(args)
  File "run_realworld/run.py", line 80, in main
  tgt_masks = inference_one_image(np.array(tgt_img_PIL), grounded_dino_model, sam_predictor, box_threshold=cfgs['box_threshold'], text_threshold=cfgs['text_threshold'], text_prompt=obj, device="cuda").cpu().numpy() # you can set point_prompt to traj[0]
  File "/home/dr/Desktop/Gaurav/Research/BU/RAM_code/vision/GroundedSAM/grounded_sam_utils.py", line 152, in inference_one_image
  boxes_filt, pred_phrases = get_grounding_output(
  File "/home/dr/Desktop/Gaurav/Research/BU/RAM_code/vision/GroundedSAM/grounded_sam_utils.py", line 72, in get_grounding_output
  model = model.to(device)
  File "/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
  return self._apply(convert)
  File "/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
  module._apply(fn)
  File "/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
  module._apply(fn)
  File "/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
  module._apply(fn)
  [Previous line repeated 3 more times]
  File "/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
  param_applied = fn(param)
  File "/home/dr/anaconda3/envs/ram/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
  return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 7.75 GiB total capacity; 6.05 GiB already allocated; 1.46 GiB free; 6.20 GiB allowed; 6.19 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[yxKryptonite]

Hi, I think the problem is that your GPU memory is insufficient to run Grounded-SAM. Could you provide more details of your GPU type and memory?