AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; adaptive factual-guided activation editing targets object hallucination categories

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AFTER is an inference-time activation editing method that mitigates object hallucination in Large Vision-Language Models (LVLMs) by steering internal activations toward factual textual semantics derived from ground-truth annotations, replacing the degraded-visual approach of prior work. It comprises two components: Factual-Augmented Activation Steering (FAS), which constructs a general positive steering vector from factual descriptions, and Query-Adaptive Offset Optimization (QAO), which personalizes that vector per query via a lightweight MLP estimator. On AMBER, it achieves up to a 16.3% reduction in hallucination over the baseline.

Problem¶

LVLMs exhibit object hallucination across three types—category, attribute, and relation—primarily driven by language bias, where the model prioritizes textual priors over visual evidence. Existing inference-time activation editing methods (e.g., VTI, ICT) construct steering vectors by contrasting activations from original images against activations from degraded/perturbed images. This strategy operates entirely within the visual domain and thus (1) cannot leverage positive factual textual guidance to explicitly counter visual-textual disparity, and (2) produces a single, query-agnostic steering vector that cannot adapt to the distinct visual semantics emphasized by different queries.

Method¶

AFTER computes a two-stage activation edit applied to the top-$K$ attention heads (those with largest vector magnitudes) of LVLM $\mathcal{M}$:

\[h^{l+1} = h^l + \text{Concat}^H_{k=1}\!\left(z^{l,k} + \alpha \cdot \left[\mathcal{G}(z^{l,k}) + \bar{d}\right]\right) \cdot W^l_o\]

FAS (Factual-Augmented Activation Steering): Starting from 500 COCO training images with full ground-truth annotations (bounding boxes, segmentation, category labels), FAS extracts three fact types: - Category facts $\mathcal{T}_c$: directly from category labels. - Attribute facts $\mathcal{T}_a$: color (dominant pixel in segmentation mask), shape (contour analysis of polygons), count (label frequency). - Relation facts $\mathcal{T}_r$: spatial relations (left/right/overlapped) from bounding-box center offsets and IoU.

These are textualized by a helper LVLM $\mathcal{F}$ into a factual description $t^+ = \mathcal{F}(\mathcal{I}_{fst}; (x, [\mathcal{T}_c, \mathcal{T}_a, \mathcal{T}_r]))$. Trusted-untrusted pairs $\langle(t^+, q_i), (x, q_i)\rangle$ yield activation pairs $\langle z^+_i, z_i\rangle$. The general steering vector is: $$\bar{d} = \frac{1}{n \cdot |\mathcal{X}|} \sum_{\mathcal{X}} \sum_{i=1}^n (z^+_i - z_i)$$

QAO (Query-Adaptive Offset Optimization): For each query $q_i$, QAO extracts object categories $\{q_{i,j}\}$ mentioned in $q_i$, retrieves or negates their sub-descriptions from $t^+$ to form a query-focused factual description $t^*_i$, and computes the query-specific vector $\tilde{d}_i = z^*_i - z_i$. A single-layer MLP offset estimator $\mathcal{G}$ is trained to predict the offset $o_i = \tilde{d}_i - \bar{d}$ from the current activation $z_i$ via MSE loss: $$\mathcal{L}_\mathcal{G} = \frac{1}{n \cdot |\mathcal{X}|} \sum_\mathcal{X} \sum_{i=1}^n \|\mathcal{G}(z_i) - o_i\|^2$$

At inference, the adaptive edit $\alpha \cdot [\mathcal{G}(z^{l,k}) + \bar{d}]$ is added to the top-$K=64$ heads with $\alpha=7$.

Key Contributions¶

FAS: First activation-editing method to use positive factual textual semantics (from ground-truth annotations) as the trusted steering direction, explicitly modeling visual-textual associations for all three hallucination types.
QAO: Lightweight single-layer MLP offset estimator that converts a general steering vector into a query-specific one without fine-tuning the LVLM backbone.
Generalization: Evaluated across architecturally distinct LVLMs (projection-based LLaVA-v1.5 and Shikra; query-based InstructBLIP), showing consistent gains.
Efficiency: No LVLM fine-tuning; QAO training is decoupled from model; helper LVLM $\mathcal{F}$ is only used offline.

Results¶

AMBER (generative hallucination):
LLaVA-v1.5: Hal ↓ from 31.6 (baseline) → 20.5 (AFTER); best prior method VTI: 23.7 → 16.3% relative reduction over baseline
InstructBLIP: Hal ↓ 35.4 → 25.1 (vs. VTI: 26.7)
Shikra: Hal ↓ 49.5 → 33.2 (vs. VTI: 38.5)
POPE (discriminative):
Average +4.1% accuracy and +2.6% F1 over baselines; surpasses SOTA ICT by +1.3% accuracy and +0.9% F1
MME hallucination subset: Gains on all four sub-tasks (existence, count, position, color) across all three models; e.g., LLaVA-v1.5 count score improves from 123.3 (baseline) to 138.3 (AFTER vs. ICT's 126.7)
Ablation: Removing QAO (w/o QAO) consistently hurts all metrics, confirming its additive value over FAS alone.

Limitations¶

Annotation dependency: FAS requires pixel-level segmentation masks (for color) and bounding boxes (for relations), restricting offline vector construction to richly annotated datasets like COCO.
Color annotation is manual: Color attributes are derived from pixel statistics, which is fragile for multi-colored or occluded objects.
Helper LLM cost: Textualization of facts requires a capable LVLM $\mathcal{F}$ (GPT-4o or GPT-4o-mini tested), adding offline compute.
Fixed head selection: Top-$K$ heads are chosen by vector magnitude rather than task-adaptive criteria; optimality of this heuristic is not systematically justified.
Evaluation scope: Benchmarks (POPE, MME, AMBER) are COCO-derived, so the method and evaluation share the same data distribution.

Relevance to Vision-Language Models¶

AFTER addresses a mechanistic root cause of VLM failure—language bias manifesting in internal attention activations—rather than patching outputs through decoding or retraining. The insight that factual textual semantics provide a stronger positive steering signal than degraded visual counterparts is a meaningful theoretical advance over VTI and ICT, directly relevant to researchers studying VLM internal representations and interventions. The query-adaptive offset mechanism is a lightweight but principled way to handle the diversity of visual-textual associations without expanding the model, making it practically attractive for deployment. For the VLM hallucination sub-field, AFTER sets a new precedent: activation editing should leverage multimodal factual grounding rather than staying within the visual perturbation paradigm.