Skip to content

Re-Aligning Language to Visual Objects with an Agentic Workflow

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Real-LOD is a cyclic agentic workflow that reduces VLM hallucinations in automatically generated language-object training pairs by iteratively applying planning, tool use, and reflection to re-align object descriptions with visual content. Using only 0.18M re-aligned images, a standard LOD model trained on the resulting dataset surpasses prior methods by ~50% on standard benchmarks. The key insight is that preserving VL alignment quality during data scaling is as critical as scaling quantity.

Problem

VLMs used to auto-generate training captions for language-based object detection (LOD) frequently hallucinate: small objects are misdescribed (VLMs are trained on global image captions), and generic text prompts cause attribute/relation errors. Appending these hallucinated expressions to training data degrades vision-language alignment and stalls LOD performance as data scales.

Method

Real-LOD wraps a fine-tuned LLM agent (Real-Agent, ChatGLM-6B + LoRA) in a three-step cyclic loop over each flagged expression:

  1. Planning (neural symbolic): Real-Agent receives the raw expression, image caption, object bounding box, category, and reflector feedback. It classifies the expression into one of 5 predefined states and dispatches the matching action:
  2. State 1 (Right) → Stop
  3. State 2 (Wrong) → Rewrite via LLM (Vicuna-v1.5-13B)
  4. State 3 (uncertain: category/attribute) → VLM with object crop
  5. State 4 (uncertain: relation/accessory) → VLM with extended crop

  6. State 5 (uncertain: location/behavior) → VLM with red-box object highlight

  7. Tool use: Real-Agent dynamically sets Prompt and Image editing parameters for the tool pool (LLaVA-v1.6-34B for VLM, Vicuna-v1.5-13B for LLM). Visual context (cropped/highlighted crops) and customized prompts feed the tool models to obtain targeted re-descriptions.

  8. Reflection: A separate Vicuna-v1.5-13B reflector reads the tool output alongside the image caption, judges correctness, and returns structured textual feedback to Real-Agent, closing the loop.

SigLIP (\(\text{score} < 0.5\)) pre-filters roughly 75% of data before the workflow, so only ~25% (misaligned pairs) pass through Real-LOD.

Key Contributions

  • A complete domain-specific agentic workflow (planning → tool use → reflection, cyclic) for VL data curation in LOD, designed from an empirical taxonomy of 6 hallucination categories.
  • Neural symbolic action design: 5 pre-specified states/actions that map to concrete, interpretable image and prompt edits, removing ambiguity from LLM action generation.
  • Real-Data: 0.18M images / 1.4M re-aligned language-object pairs constructed at low cost.
  • Empirical demonstration that data alignment quality is independently important from data quantity for LOD scaling.

Results

  • Real-Model (Swin-B backbone, trained on Real-Data) surpasses existing LOD methods by ~50% on standard benchmarks (RefCOCO/g/+, OmniLabel, Ref-L4).
  • Real-LOD successfully re-aligns the majority of category and attribute hallucinations (highest two categories by frequency, per Fig. 6–7).
  • Only 15k fine-tuning examples are needed to train Real-Agent (via LoRA on ChatGLM-6B).

Note: The paper reports relative ~50% improvement; absolute numbers on individual benchmarks are present in the paper's tables but are not reproduced in the provided excerpt.

Limitations

  • The 5 states and corresponding image edits are hand-designed based on data analysis from a single dataset (Objects365); coverage of out-of-distribution hallucination types is uncertain.
  • Tool pool is fixed (LLaVA-v1.6-34B + Vicuna-v1.5-13B); swapping to newer or weaker models would require re-analysis.
  • SigLIP pre-filtering with a fixed threshold (0.5) may leave some misaligned pairs uncorrected or filter correctly-described-but-low-similarity pairs.
  • Cyclic overhead per expression is significant; wall-clock cost at scale is not analyzed in the excerpt.
  • Real-Agent training requires manual state annotation for 15k samples, limiting automated bootstrapping.

Relevance to Harnesses / Meta-Harnesses

Real-LOD is a concrete domain-specific meta-harness: a fixed orchestration layer (cyclic plan–act–reflect) that routes sub-tasks to specialist tool models (VLM, LLM, image editor) based on a state classifier. The neural symbolic design—pre-specifying states and actions rather than letting the LLM plan freely—is a reusable harness pattern for scenarios where the action space is enumerable and well-understood. The SigLIP-as-router pattern (pre-filter data before invoking the expensive agentic loop) exemplifies staged harness design, relevant to anyone building multi-model pipelines where selective invocation controls cost. The reflection step as an independent LLM judge (rather than self-evaluation) is directly applicable to quality-control harnesses for data curation.