Seeing What’s Wrong: A Trajectory-Guided Approach to Caption Error Detection

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; trajectory-guided multi-score method detects subtle image-caption errors

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TRACED introduces caption trajectories — ordered sequences of iteratively edited captions that maximize an image-text relevance score — as richer signals for detecting mislabeled image-caption pairs. The core insight is that correct captions stabilize under iterative refinement while erroneous captions improve substantially, a pattern that discriminates errors more reliably than any single alignment score. Beyond detection, the trajectory also localizes specific erroneous tokens, enabling guided VLM-based caption correction.

Problem

Existing image-caption error detection assigns a single similarity score per pair (CLIP cosine sim, BLIP ITM probability, LEMoN neighborhood aggregation). This is fundamentally limited: captions with subtle errors (wrong negation, incorrect color, mislabeled object) can still score above threshold, while correct but imprecisely worded captions can score below it. The field also lacks a realistic fine-grained noise benchmark — standard synthetic noise (random caption swaps, noun-matched swaps) produces obviously mismatched pairs that are easy to detect.

Method

TRACED wraps any existing alignment scoring function \(s: \mathcal{X} \times \mathcal{Y} \rightarrow \mathbb{R}\) into a trajectory construction loop:

1. Trajectory generation: Starting from the original caption \(x_0\), at each of \(T\) steps generate \(N\) candidate alternative captions and select the one maximizing \(s(x_t, y)\), yielding the sequence \(x_0, x_1, \ldots, x_T\).

2. Trajectory evaluation: Construct a feature vector $\(e(x_0,\ldots,x_T,y) = [s(x_0,y),\ldots,s(x_T,y),\; c(x_1,x_0),\ldots,c(x_T,x_0)]\)$ where \(c(x_t, x_0)\) is the cosine similarity between embeddings of the edited and original captions, capturing semantic drift.

3. Candidate exploration strategies (three variants):

4. Elimination (Elim): generate \(L\) candidates by deleting one token at a time; \(O(L)\) forward passes, no gradients.
5. Greedy Coordinate Descent (GCD): gradient-guided top-\(K\) token replacements for each of \(L\) positions; sample \(N\) from the \(KL\) pool.

6. Fast GCD (FGCD): first run Elimination to find the single most impactful token, then explore only top-\(K\) replacements for that token; requires one gradient computation and \(K + L\) forward passes.

7. Classification: trajectory embeddings are used as features for an XGBoost/CART classifier trained to predict correct vs. erroneous pairs.

8. Interpretability / correction: tokens whose removal most increases \(s\) are flagged as likely erroneous. These flags are injected as hints into VLM prompts (InternVL3) for iterative caption correction.

Key Contributions

• TRACED framework: trajectory-based, model-agnostic wrapper that augments any alignment scorer for error detection.
• Three exploration algorithms: Elimination (cheapest), GCD (most expressive), FGCD (hybrid, one gradient step).
• Fine-grained synthetic noise benchmark: GPT-4o-mini generates minimally but semantically meaningfully perturbed captions (e.g., "no jacket" vs. "black jacket"), better reflecting real annotation errors than caption-swap baselines.
• Interpretable token-level error localization via trajectory inspection.
• VLM correction application: TRACED hints improve caption correction; InternVL3-1B fine-tuned on TRACED-guided InternVL3-14B outputs (InternVL3-1B-FT) tested for scalable cleaning.

Results

• TRACED improves detection accuracy over baselines by:
• Up to 2.5% on MS COCO (averaged over random, noun, fine-grained noise at 50% noise rate)
• Up to 2.8% on Flickr30k
• Up to 2.4% on MM-IMDb (random + noun noise only)
• With TRACED-BLIP(ITM) + Elimination on Flickr30k/MS COCO: 89.5% ± 0.2 accuracy, +1.3% ± 0.2 over BLIP(ITM) baseline; Elim outperforms FGCD (+0.8%) and GCD (+0.3%).
• Caption correction: TRACED-guided prompting of InternVL3 yields up to +14.5% BLIP-alignment score improvement over unguided correction.
• Throughput: TRACED-BLIP (Elim) classifies 1M pairs in ~6.5 hours on 4× NVIDIA L40 40GB GPUs.

Limitations

• Accuracy improvements are modest in absolute terms (2–3%), which may not justify overhead for all use cases.
• Fine-grained noise evaluation restricted to Flickr30k and MS-COCO due to ChatGPT API cost.
• GCD and FGCD require gradient access through the scoring function, limiting applicability to differentiable scorers.
• Trajectory construction adds wall-clock overhead proportional to trajectory length \(T\) and exploration breadth \(N\); the paper reports only BLIP-ITM throughput — CLIP/LEMoN costs are not quantified in the excerpt.
• The classifier (XGBoost/CART) requires labeled trajectory data for training; performance in fully unsupervised settings is not evaluated.

Relevance to Vision-Language Models

TRACED directly addresses a root-cause problem in VLM training: noisy image-caption supervision degrades alignment and introduces biases. By enriching the error detection signal from a scalar score to a multi-step trajectory, it provides a drop-in data-cleaning layer compatible with CLIP, BLIP, and LEMoN — the exact models used for VLM pretraining filtering. The new fine-grained noise type (subtle semantic perturbations via GPT-4o-mini) is more representative of real web-scraped annotation noise than caption swaps, making TRACED's evaluation directly relevant to practitioners cleaning datasets like LAION or CC12M. The token-level interpretability also opens a path toward targeted, scalable VLM-assisted caption repair, relevant to synthetic data curation pipelines.