Skip to content

Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Reflexive Guidance (ReGuide) is a two-stage, model-agnostic prompting strategy that improves out-of-distribution detection (OoDD) in large vision-language models (LVLMs) by having the model first generate image-adaptive auxiliary OoD class concepts, then use those concepts as negative class candidates in classification. The paper also establishes the first systematic OoDD benchmark framework for generative LVLMs, revealing that proprietary models substantially outperform open-source ones, and that open-source models exhibit pathologically biased confidence scores (100% FPR).

Problem

OoDD has been extensively studied for single-modal classifiers and CLIP-style models, but the OoDD capability of generative LVLMs (GPT-4o, Gemini, Claude, InternVL2) expressed through natural language confidence responses is almost entirely unstudied. Existing VLM confidence literature does not specifically address the task of distinguishing in-distribution (ID) from OoD inputs; a principled evaluation framework and improvement method are both missing.

Method

Evaluation framework: A structured prompt is designed with four components—task description, a named rejection class ("none of these classes"), formatting guidelines addressing known failure modes (score-prediction mismatches, all-zero confidence, missing scores), and two-shot in-context examples (one ID, one OoD). The OoD score is the softmax-normalized maximum confidence over ID classes; low scores indicate OoD.

ReGuide (two stages): 1. Stage 1 — image-adaptive concept generation: Given the input image, prompt the LVLM to generate 2N class names: N visually similar (near-OoD proxies, A^near_aux) and N visually dissimilar (far-OoD proxies, A^far_aux). 2. Stage 2 — OoDD with auxiliary classes: Append all 2N suggested concepts as auxiliary OoD classes alongside the original ID class set Y. The same OoD score (max confidence over original ID classes) is used. The rejection class is retained as a fallback.

The approach requires no external data, no fine-tuning, and the same prompt template works across different LVLMs.

Key Contributions

  • First systematic OoDD evaluation framework for generative LVLMs using structured natural-language confidence elicitation.
  • Comprehensive benchmark of proprietary (GPT-4o, Claude 3.5 Sonnet, Gemini Pro 1.5) and open-source (LLaVA-v1.6, GLM-4v-9B, InternVL2-26B/76B) models on ImageNet200 and CIFAR10 OoDD benchmarks.
  • Empirical finding that most open-source LVLMs yield 100% FPR due to severely biased confidence values; visual feature interpretability correlates with OoDD capability.
  • ReGuide: a training-free, model-agnostic, image-adaptive self-prompting method that generates negative concepts from the model itself.
  • Analysis of class order sensitivity, scaling laws with model size, confidence calibration (ECE/AURC), and failure mode taxonomy.

Results

  • Baseline ranking (ImageNet200, AUROC ID vs. all OoD): GPT-4o 88.50% > Gemini Pro 1.5 85.74% > GLM-4v 77.86% > InternVL2-76B 74.14% > Claude 3.5 Sonnet 73.69%; LLaVA-v1.6 and InternVL2-26B yield 50% AUROC (random-chance).
  • Open-source FPR: LLaVA-v1.6, GLM-4v, InternVL2-76B all achieve 100% FPR@95%TPR at baseline.
  • ReGuide on InternVL2-26B: AUROC improves from 63.11% → 93.19% (all OoD); near-OoD AUROC (NINCO) from 58.31% → 86.53%; ID accuracy 91.23% → 93.53%.
  • ReGuide on InternVL2-76B: AUROC improves from 75.30% → 90.60% (all OoD); near-OoD AUROC (NINCO) from 71.89% → 91.35%.
  • ReGuide on GPT-4o: AUROC improves from 89.85% → 94.61% (all OoD), especially near-OoD (NINCO AUROC 93.96% → 96.76%).
  • OoD prediction rate: ReGuide raises InternVL2-26B's rate of OoD inputs predicted as non-ID classes from 26.88% to 89.13%.
  • Proprietary models generally match or exceed single-modal SOTA OoDD methods (SCALE, fDBD, AugMix+ASH) without any task-specific training.

Limitations

  • ReGuide doubles inference cost (two LVLM calls per image), which is non-trivial for large proprietary models at scale.
  • Evaluated on only 25% and 5% subsets of benchmarks due to API rate limits and cost; full-scale results may differ.
  • For larger models (InternVL2-76B, GPT-4o), ReGuide improves AUROC but degrades FPR@95%TPR for a small subset of ambiguous ID images where the model's strong visual recognition causes misclassification into suggested auxiliary classes.
  • Stage 1 concept quality depends on the LVLM's own visual interpretation ability; models with poor image understanding (LLaVA-v1.6, GLM-4v) are excluded from ReGuide experiments.
  • The framework requires confidence scores as structured output; models with low instruction-following ability (e.g., QWEN-VL-Chat, <1% valid rate) are unusable.

Relevance to Vision-Language Models

This paper directly probes a critical but underexplored reliability dimension of LVLMs: whether their natural-language confidence outputs are trustworthy enough to support OoD rejection in deployment. The finding that open-source LVLMs produce degenerate confidence distributions (100% FPR) despite strong classification performance exposes a fundamental gap between task accuracy and calibrated uncertainty—relevant to any safety-critical VLM application. ReGuide demonstrates that self-guided prompting using the model's own visual understanding is a viable, training-free path to improving OoDD, connecting to broader work on chain-of-thought prompting and negative-label augmentation in CLIP-based OoDD. The sensitivity to prompt class ordering and the positive-label bias attributed to RLHF alignment are important findings for VLM fine-tuning research.