Skip to content

Unified Reinforcement and Imitation Learning for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RIL (Unified Reinforcement and Imitation Learning) is a training framework that teaches small VLMs (1B–8B) to mimic large teacher VLMs (72B–78B) by combining GRPO-based RL with adversarial imitation learning via a trainable LLM-based discriminator. The discriminator provides a binary similarity reward (teacher vs. student response style) while LLM-as-a-Judge supplies a binary answer-correctness reward; together these guide the student without requiring architectural changes or chain-of-thought reasoning. RIL-trained 7B/8B models close much of the gap to—and sometimes exceed—closed-source VLMs such as GPT-4o and Claude-3.5-Sonnet on 14 vision-language benchmarks.

Problem

Large VLMs achieve top performance but are impractical on resource-constrained devices (mobile, AR). Existing small-model approaches either require architectural modifications, lengthy "think-answer" chain-of-thought inference, or logit/feature-level distillation that demands identical tokenizers and image encoders between teacher and student. Pure RL with GRPO also struggles when the student has insufficient domain knowledge to produce any correct rollouts, limiting its training signal.

Method

RIL integrates GRPO (specifically Dr.GRPO for unbiased advantage estimation) with GAIL-style adversarial imitation:

  1. Discriminator pre-training: A discriminator \(D_\phi\) is initialised with the same architecture and weights as the student VLM but with the language head replaced by a linear sigmoid head (d→1). It is trained on cached student/teacher responses to output 1 for student outputs and 0 for teacher outputs.

  2. Reward design: For each student response \(o_i\) the total reward is the sum of two binary signals:

  3. Similarity reward: \(\mathbf{1}[D_\phi(q, o_i) < 0.5]\) — low discriminator score means the response resembles teacher style.

  4. Answer reward: LLM-as-a-Judge (Qwen2.5-32B) scoring factual correctness against ground truth (0 or 1).

  5. RIL loop: Each iteration (i) updates \(D_\phi\) on the current mixed batch of student and teacher responses, then (ii) updates the student policy \(\pi_\theta\) via the Dr.GRPO clipped-ratio objective using the composite reward. Teacher responses are included in the GRPO rollout set to provide direct exemplars of correct answers, stabilising training when the student alone cannot reach correct solutions.

  6. Multi-teacher extension: Responses from two heterogeneous large VLMs (Qwen2.5-VL-72B + InternVL3-78B) are pooled, increasing response diversity and discriminator robustness.

  7. Similarity rewards are binarised (0/1) rather than left continuous; ablations confirm this yields cleaner learning signals.

Key Contributions

  • RIL framework: novel combination of Dr.GRPO and GAIL for VLM compression, operating purely on generated natural-language text with no feature-level constraints.
  • LLM-as-a-Judge integration: enables reward computation for open-ended VQA tasks where rule-based parsing fails (e.g., "How to cook this dish?").
  • Multi-teacher training: pooling responses from architecturally diverse large VLMs demonstrably outperforms single-teacher RIL.
  • Inference efficiency: no think-answer overhead; trained models run at the same latency as the base student VLM.
  • Distillation synergy: RIL shows amplified gains on VLMs already pre-aligned via feature distillation (MiniLLM, DistilLLM, LLaVA-KD, VLsI).

Results

  • Qwen2.5-VL-7B + RIL (multi-teacher): +7.6 pp on MathVista (67.8→79.7), +8.1 pp on ChartQA (87.3→95.6), +13.6 pp on BLINK (56.4→70.0), +9.8 pp on MM-Vet (71.8→80.4) vs. base; outperforms Dr.GRPO alone on all 14 benchmarks.
  • InternVL3-8B + RIL (multi-teacher): ChartQA 95.3, MMB 88.1, MathVista 77.8, MMStar 74.8.
  • vs. closed-source: Qwen2.5-VL-RIL-7B matches or exceeds GPT-4o on ChartQA (95.6 vs. —), BLINK (70.0 vs. 68.0), SEED (80.5 vs. 77.1); competitive with Claude-3.5-Sonnet and Gemini-1.5-Pro across most benchmarks.
  • vs. same-scale open-source: RIL-7B outperforms all listed 7B–8B open-source models on ChartQA, MathVista, MMB, and SEED.
  • Hyperparameter ablation: µ=1 update per cycle is optimal; β=0.01 KL penalty best; self-attention and word-embedding layers matter most; binary similarity reward beats continuous and multi-level discretisations.
  • Multi-teacher gain (3B example): +6.5 pp on MathVista and +13.1 pp on BLINK over single-teacher RIL.
  • Discriminator necessity: fixed pre-trained discriminator already beats RL-only baselines, but continuously updated discriminator delivers highest final performance.

Limitations

  • RIL training is compute-intensive: discriminator pre-training on 1.2M samples takes 1–3 days on 256 A100s; SFT takes 3–5 days; the RIL loop adds 3–5 more days on 8 A100s.
  • Current application is restricted to the post-SFT alignment phase; the authors acknowledge extending it to the visual instruction tuning stage as future work.
  • Teacher responses must be pre-generated and cached for efficiency; quality and diversity are bounded by the teacher models chosen.
  • LLM-as-a-Judge (Qwen2.5-32B) introduces an additional inference cost and may itself be imperfect on highly ambiguous or culturally specific answers.
  • The discriminator architecture must mirror the student at initialisation, creating a tight coupling between discriminator and student model family.

Relevance to Vision-Language Models

RIL directly addresses the practical deployment bottleneck of VLMs by demonstrating that small (1B–8B) models can approach closed-source frontier performance through training-time behaviour alignment rather than scale or architectural augmentation. The adversarial imitation framing is a principled alternative to logit distillation and avoids the cross-architecture tokenizer alignment problem, making it readily applicable to any student–teacher pair sharing only a natural-language interface. The dual-reward design (style similarity + factual correctness via LLM-as-a-Judge) is a general recipe that extends RL-from-feedback methods to open-ended multimodal QA beyond math/grounding niches. For the VLM community, RIL establishes a replicable post-training pipeline that stacks on top of existing distillation-based VLMs and preserves inference efficiency.