Skip to content

ESC: Emotional Self-Correction for Reliable Vision-Language Models

🕒 Published (v1): 2026-07-01 14:25 UTC · Source: Arxiv · Venue: ECCV · link

Why this paper was selected

Emotional self-correction for VLMs without post-training; lightweight reliability fix applicable broadly

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ESC (Emotional Self-Correction) is a training-free inference-time framework that injects emotionally-framed feedback into VLMs to trigger self-correction of incorrect initial responses. A separate verifier VLM detects unreliable responses and selects the emotional cue; the target model then revisits its reasoning under the affective shift. ESC achieves consistent improvements across safety, hallucination, vision-centric perception, and multimodal reasoning benchmarks without modifying model weights.

Problem

VLMs remain vulnerable to hallucination and unsafe outputs, and existing self-correction methods either require expensive post-training (RL, supervised fine-tuning on reflective trajectories) or depend on high-quality corrective feedback that is hard to obtain at scale. Intrinsic self-correction is additionally limited by the self-correction blind spot: models can correct others' errors but systematically struggle to detect and revise their own.

Method

ESC operates in three stages at inference time, with no weight updates:

  1. Initial generation: Target VLM \(M_T\) generates \(R_\text{initial}\) from image-question pair \((I, Q)\).
  2. Verification: A separate verifier VLM \(M_V\) assesses \(R_\text{initial}\). If it judges the response unreliable, it selects an emotional feedback signal \(F_\text{emotional}\) (drawn from negative-valence quadrants of Russell's Circumplex Model of Affect — chosen because negative-valence prompts yield the largest behavioral shift across models).
  3. Emotional revision + final selection: \(F_\text{emotional}\) is prepended to a follow-up query to \(M_T\), producing \(R_\text{revised}\). The verifier then compares \(R_\text{initial}\) and \(R_\text{revised}\) and returns the better one.

The framework avoids triggering revision when \(R_\text{initial}\) is already accepted, reducing unnecessary compute. The verifier is kept separate from the target to circumvent the self-correction blind spot; even 3–4B verifiers (Qwen3-VL-4B, Qwen2.5-VL-3B, Gemma3-4B) achieve performance close to the 12B default.

The emotional cue selection is motivated by two preliminary empirical findings on 5 VLMs (LLaVA-1.5-7B, LLaVA-1.6-7B, Qwen2-VL-7B, InternVL2.5-8B, Pixtral-12B) on VLSafe: (1) any emotional cue reduces attack success rate (ASR) by 7.2–26.4 pp; (2) negative-valence cues are the strongest behavioral regulators.

Key Contributions

  • Empirical finding: Systematic evidence that VLMs perceive emotional cues and use them for self-correction, with negative-valence cues producing the largest and most consistent behavioral shift.
  • ESC framework: A plug-and-play, training-free verify-then-emotionally-revise pipeline using a separate verifier to avoid the self-correction blind spot.
  • Ablation establishing causality: Controlled comparisons against verifier-only (VO), ZeroCoT, Self-Refine (SR), corrective prompting (Corr), and psychological prompting (Psych) isolate emotional content as the primary driver of improvement.
  • Cautiousness analysis: Thinking-trace scoring via Gemma-4-26B shows ESC raises mean cautiousness from 3.31 (baseline/VO) to 4.50, with median 5.0, confirming a qualitative shift in reasoning behavior.
  • Generalization: Demonstrated on LLaVA-1.5-7B, Qwen2-VL-7B, and newer Qwen3-VL-8B and InternVL3-8B across four benchmark families.

Results

Safety: - VLSafe ASR: LLaVA-1.5-7B 71.6% → 25.3% (−46.3 pp); Qwen2-VL-7B 20.0% → 9.9% (−10.1 pp) - MMSafetyBench: ASR reductions across all 13 scenario categories for both models

Hallucination (POPE / HallusionBench): - Qwen2-VL-7B POPE Adversarial Acc: 50.80% → 80.43% (+29.63 pp); F1: 3.40 → 76.17 (+72.77 pp) — recovering collapsed inference behavior - LLaVA POPE: steady +1.06–2.08 pp across splits; HallusionBench aAcc +1.06

Multimodal reasoning (MM-Vet, MathVista, MMStar, MMMU, AI2D): - LLaVA: +1.08 (MM-Vet), +1.20 (MathVista), +2.49 (AI2D); no benchmark degraded - Qwen2: gains marginal on knowledge-intensive tasks; +1.55 on AI2D

Vision-centric perception (MMVP, RealWorldQA, BLINK): - MMVP pair accuracy: LLaVA +7.33 pp; Qwen2 +4.00 pp - RealWorldQA: Qwen2 +14.51 pp Acc

Ablation (VLSafe, LLaVA-1.5-7B target): - Baseline 71.6%, VO 69.1%, ZeroCoT 70.1%, SR 49.3%, Corr 48.6%, Psych 54.4%, ESC 31.2%

Newer models (VLSafe): - Qwen3-VL-8B: 8.4% → 3.2% (−5.2 pp); InternVL3-8B: 10.5% → 6.5% (−4.0 pp)

Limitations

  • Introduces additional inference cost: two model calls per sample (initial + revised) plus verifier calls; selective verification mitigates but does not eliminate this overhead.
  • Gains on stronger or more safety-aligned models (Qwen2-VL-7B, Qwen3-VL-8B) are smaller in absolute terms, suggesting diminishing returns as baseline alignment improves.
  • The verifier itself must be a separate model; using the target model as its own verifier limits improvement (LLaVA self-verifying: ASR only drops to 50.3% vs. 31.2% with external verifier).
  • Experiments are limited to two primary target models (LLaVA-1.5-7B and Qwen2-VL-7B); broader validation across model families and scales is not reported in the main results.
  • The mechanism by which emotional cues activate latent self-correction is empirically demonstrated but not theoretically characterized; the paper does not explain why the VLMs' internal representations respond to affective framing.
  • Emotional cue selection strategy (negative-valence preference) is heuristic and architecture-dependent (e.g., Pixtral-12B shows minimal sensitivity to positive-valence cues).

Relevance to Vision-Language Models

ESC directly addresses a core reliability challenge in VLMs — hallucination and unsafe outputs — through a novel inference-time mechanism that requires no retraining, making it immediately applicable to frozen or API-only models. The finding that emotional prompting shifts VLM reasoning toward greater cautiousness (measurably in thinking traces) opens a new axis of behavioral control orthogonal to RLHF/DPO alignment, relevant to anyone studying inference-time scaling or steering of VLMs. The work also provides evidence that VLMs have latent self-correction capacity that can be externally triggered, reinforcing the broader line of research on emergent reasoning behaviors in multimodal models. For VLM researchers, the systematic characterization of emotional cues using Russell's circumplex model provides a principled framework for studying affective inputs as a class of control signals.