Skip to content

GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GuardReasoner-VL is a reasoning-based guard model for VLMs that performs deliberative chain-of-thought reasoning before issuing a harmful/unharmful verdict on multimodal inputs and outputs. It is trained via SFT on a curated 123K-sample reasoning corpus followed by online GRPO-based RL with three tailored components: safety-aware data concatenation, dynamic clipping, and a length-aware reward. It outperforms the runner-up by 19.27% average F1 across multimodal guardrail benchmarks.

Problem

Existing VLM guard models (e.g., LLaMA Guard 3-Vision, Beaver-Guard-V) output only classification labels without justification, lack generalization to unseen harm categories, and are restricted to offline training. Direct safety alignment of the victim VLM incurs an "alignment tax" degrading helpfulness and reasoning. A reasoning-based guard model that is interpretable, generalizable across modalities (text, image, text-image), and token-efficient is absent.

Method

Data curation. GuardReasoner-VLTrain is built from existing text-only safety corpora (WildGuardTrain, AegisTrain, BeaverTails, ToxicChat), image-only datasets (UnsafeBench, BadNews, HatefulMemes, HOD), and text-image pairs (SPA-VL-Train). GPT-4o generates reasoning chains for each sample, yielding 123K samples with 631K reasoning steps.

Cold-start SFT (R-SFT). The base model is fine-tuned to jointly generate a reasoning process R and moderation label Y given the guardrail instruction, user prompt, and VLM response.

Online RL (GRPO-based). Three techniques refine the SFT model: 1. Safety-aware data concatenation: hard samples (all predictions wrong under rejection sampling) are augmented by concatenating pairs of text/image inputs; the new label is harmful if any constituent is harmful, forcing detection of harm buried in benign context. 2. Dynamic clipping: the GRPO clip threshold \(B_s = \frac{s_\text{total}-s}{s_\text{total}} \cdot \epsilon\) decreases from large (exploration) to small (exploitation) over training steps. 3. Length-aware safety reward: \(r = \frac{-1 + r_\text{safety}}{\min(l_\text{norm}, \beta)^2}\), where \(r_\text{safety} \in [-1,0]\) encodes format + correctness, \(l_\text{norm}\) is normalized reasoning length, and \(\beta\) is a cut-off preventing over-thinking. Incorrect predictions are incentivized to extend reasoning; correct predictions are penalized for unnecessary length.

GuardReasoner-VL-Eco uses \(\beta=16\) to trade ~1–1.4% F1 for ~12–14% fewer output tokens.

Key Contributions

  • GuardReasoner-VL: a reasoning-first VLM guard model covering text, image, and text-image modalities.
  • GuardReasoner-VLTrain: 123K-sample multimodal reasoning corpus with 631K GPT-4o-generated reasoning steps.
  • Safety-aware data concatenation for augmenting hard, diverse training samples for online RL.
  • Dynamic clipping parameter schedule balancing exploration/exploitation in GRPO.
  • Length-aware safety reward jointly optimizing accuracy, format compliance, and token efficiency.
  • Public release of data, code, and 3B/7B model checkpoints.

Results

  • +19.27% average F1 over the runner-up across multimodal guardrail benchmarks (Figure 1).
  • Prompt harmfulness detection (Table 1, Average All): GuardReasoner-VL 7B: 79.07%; GuardReasoner-VL 3B: 78.73%; vs. LLaMA Guard 3-Vision 11B: 48.03%; Qwen2.5-VL-Instruct 7B: 56.53%.
  • Response harmfulness detection (Table 2, Average All): GuardReasoner-VL 7B: 77.58%; GuardReasoner-VL 3B: 76.56%; vs. LLaMA Guard 3-Vision 11B: 54.01%; Qwen2.5-VL-Instruct 7B: 61.58%.
  • On text-only benchmarks, performance is comparable to state-of-the-art LLM guard models (e.g., GuardReasoner 8B), despite no direct text-guard specialization.
  • Token efficiency (Table 3): Eco-3B uses 187 vs. 213 output tokens (−12.2%) with only −0.48% F1; Eco-7B uses 180 vs. 208 tokens (−13.6%) with −1.42% F1.
  • Ablation (Figure 6): each of R-SFT over plain SFT, safety-aware augmentation, dynamic clipping, and length reward individually improves F1; full model achieves 79.07% (7B) vs. 61.61% for plain SFT.

Limitations

  • Reasoning corpus labels are generated by GPT-4o, introducing potential label noise and cost at scale.
  • Safety-aware data concatenation can introduce class imbalance; the authors acknowledge this but state no imbalance was observed in practice.
  • Approximately 10–14% token overhead vs. classifier-only guards even in the Eco variant.
  • Evaluation is on fixed benchmark harm taxonomies; generalization to truly novel harm categories beyond training distribution is not quantitatively assessed.
  • Image-only SFT alone degrades textual capability, limiting single-modality fine-tuning.
  • Future extension to agentic/multi-step systems is left as future work.

Relevance to Vision-Language Models

GuardReasoner-VL directly addresses the safety infrastructure required to deploy VLMs responsibly: rather than baking alignment into the victim model (incurring the alignment tax), it provides an external, interpretable moderation layer compatible with any VLM. The reasoning-before-moderating paradigm extends the chain-of-thought RL trend (DeepSeek-R1, GuardReasoner) into multimodal safety, showing that GRPO-trained reasoning generalizes across text, image, and text-image modalities simultaneously. For VLM researchers, this establishes a strong guard baseline and a reusable 123K multimodal safety dataset, and it highlights that image-only harm (without text) remains underserved by existing guard systems.