iVGR

Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

Environment Setup

Our code is tested with Python 3.11 and CUDA 12.4.

# 1. Create and activate a clean conda environment
conda create -n ivgr python=3.11 -y
conda activate ivgr

# 2. Install the core dependencies (Transformers + vLLM)
pip install transformers==4.52.4 vllm==0.8.5.post1

# 3. Install FlashAttention from the prebuilt wheel (cu12 / torch2.6 / cp311)
wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp311-cp311-linux_x86_64.whl
pip install flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp311-cp311-linux_x86_64.whl

# 4. Install the remaining requirements
pip install -r requirements.txt

# 5. Install iVGR in editable mode (--no-deps keeps the versions pinned above)
pip install -e . --no-deps

# 6. Install the math verification utility
pip install math_verify

Data Preparation

We release the RL training data at allencbzhang/iVGR-RL-51K. The repo contains two files: gvqa_and_mathdata.parquet (the samples) and images.tar.gz (the corresponding images).

# 1. Download the dataset (parquet + image archive) from the Hugging Face Hub
huggingface-cli download allencbzhang/iVGR-RL-51K --repo-type dataset --local-dir ./data/iVGR-RL-51K

# 2. Extract the images into a folder
tar -xzf ./data/iVGR-RL-51K/images.tar.gz -C ./data/iVGR-RL-51K

After extraction, the directory should look like:

data/iVGR-RL-51K/               # ← data.image_dir
├── gvqa_and_mathdata.parquet   # training / validation samples
└── images/                     # extracted images (referenced by the parquet)

Then point the launch command in examples/train_iVGR_qwen2_5_VL.sh to these paths:

data.train_files=./data/iVGR-RL-51K/gvqa_and_mathdata.parquet \
data.val_files=./data/iVGR-RL-51K/gvqa_and_mathdata.parquet \
data.image_dir=./data/iVGR-RL-51K/ \

Note: data.image_dir is prepended to the relative image paths stored in the parquet's images column (which already include the images/ prefix), so it points to the dataset root ./data/iVGR-RL-51K/ rather than the images/ subfolder.

Training

The reward function calls an LLM judge (an OpenAI-compatible endpoint) to score answer accuracy and consistency. Set the following three environment variables at the top of examples/train_iVGR_qwen2_5_VL.sh before launching:

export JUDGE_BASE_URL=http://localhost:8000/v1   # judge endpoint (OpenAI-compatible /v1)
export JUDGE_API_KEY=EMPTY                        # API key ("EMPTY" for a local server)
export JUDGE_MODEL_NAME=judge                      # served model name (must match the server)

If you don't have an external judge API, you can host one locally with vLLM. The --served-model-name you pass here must match JUDGE_MODEL_NAME above:

vllm serve Qwen/Qwen2.5-72B-Instruct \
    --served-model-name judge \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9

Then launch training:

bash examples/train_iVGR_qwen2_5_VL.sh

Evaluation

All benchmarks are evaluated with our fork of VLMEvalKit under eval/.

VLMEvalKit needs a few extra packages on top of the training environment. Install them once inside eval/:

cd eval
pip install -r requirements.txt
pip install -e .

# Sanity-check the key versions
pip show transformers torch vllm opencv-python flash-attn

# Use the headless OpenCV build (the GUI build breaks on servers), plus dev tools
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless autopep8 ipdb

# Pin the transformers / numpy stack that VLMEvalKit is tested against
pip install transformers==4.57.6
pip install --force-reinstall numpy==1.26.4 pandas scipy scikit-learn

Then configure and run the evaluation:

# Configure the LLM judge (edit .env with your OpenAI-compatible endpoint)
cp .env.example .env

# Option A — local weights, one replica per GPU
export IVGR_CKPT_PATH=/path/to/iVGR-Qwen2.5-VL-7B
bash eval_ivgr.sh

Qualitative Results

Acknowledgements

This project is built upon EasyR1 (for training Qwen2.5-VL-7B) and veRL (for training Qwen3-VL-8B and 32B). We also sincerely thank the authors of TreeVGR for their help in reproducing their work.

BibTeX

@inproceedings{zhangivgr,
  title={iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning},
  author={Zhang, Chang-Bin and Zhong, Yujie and Zhang, Qiang and Han, Kai},
  booktitle={Forty-third International Conference on Machine Learning}
}