Skip to content

Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

Stitch and Tell (SiTe) is an annotation-free, plug-and-play data augmentation method that combats spatial hallucinations in VLMs by stitching pairs of images along a spatial axis and generating spatially-grounded captions and QA pairs from the resulting layout. It requires no human labeling, no large generative models, and no architectural changes. Evaluations across three VLM architectures and thirteen benchmarks show consistent gains on spatial benchmarks while preserving or improving general VL performance.

Problem

VLMs exhibit spatial hallucinations—incorrectly describing relative object positions—primarily because existing large-scale multimodal datasets contain very sparse spatial language (e.g., only 1.68%–7.32% of samples in CC, COCO, Flickr30K explicitly include spatial relations). The modality gap between rich visual spatial structure and impoverished spatial text prevents models from learning reliable cross-modal spatial alignment. Prior remedies (annotation-based, generative rewriting, standard augmentations like crop/rotate) are either expensive, computationally heavy, or introduce caption-image misalignment.

Method

SiTe consists of two steps applied at dataset construction time:

Image-Spatial Stitching: Two image–caption pairs (I₁, T₁) and (I₂, T₂) are concatenated on a blank canvas horizontally (left–right) or vertically (top–bottom). Two pairing strategies are offered: SiTe-rand (random pairing) and SiTe-ratio (pairs images with similar height-to-width ratios ≄1.2 for better visual balance and higher effective token density).

Text-Spatial Telling: Two forms of spatially-grounded text are generated: - Captions: Original captions T₁ and T₂ are inserted into spatial templates (35 horizontal, 29 vertical) such as "The left part shows T₁, and the right part displays T₂," making spatial relations explicit without any annotation. - QA pairs: Noun phrases are extracted from each caption (using Qwen2.5-72B-Instruct during fine-tuning; from caption text at pretraining). Overlapping nouns are removed, then entity pairs (oL, oR) are sampled to form binary spatial questions ("From the camera's perspective, is oL to the left of oR?") with answers determined automatically by construction.

Hard negatives for contrastive learning are generated by swapping spatial tokens (e.g., "left" ↔ "right") in the structured captions. Stitched pairs are removed from the raw pool to avoid duplicate supervision. The default stitching ratio λ ≈ 1/3 (stitched to raw samples) is identified as optimal via ablation. The method reduces training epoch time to 77.4% of baseline because each stitched sample encodes two images, increasing data density per step.

Key Contributions

  • Annotation-free, lightweight spatial data augmentation requiring no generative models or human labelers.
  • Two-part pipeline (Image-Spatial Stitching + Text-Spatial Telling) that injects explicit spatial supervision into both pretraining captions and fine-tuning QA data.
  • Ratio-based image pairing (SiTe-ratio) that improves visual balance and spatial supervision quality over random pairing.
  • Support for contrastive learning via automatic hard-negative generation by flipping spatial tokens.
  • 20%+ reduction in training wall-clock time as a side effect of higher sample density.
  • Empirical evaluation across 3 architectures (LLaVA-v1.5-7B, LLaVA-Qwen2-1.5B, HALVA-7B), 2 training datasets (558K, Flickr30K), and 13 benchmarks.

Results

Pretraining stage (LLaVA-v1.5-7B, 558K dataset, SiTe-ratio vs. baseline): - MMEPosition: +5.50% (133.33 vs. 127.83) - Spatial-MM: +4.19% (44.15 vs. 42.02) - COCO-QA: +1.02% (71.54 vs. 70.52) - MMBench: +0.93% (74.43 vs. 73.50)

Pretraining stage (Qwen2-1.5B, 558K, SiTe-ratio vs. baseline): - MMEPosition: +10.42% (61.67→68.00 on MMEPos score) - MMBench: +5.01% (66.68 vs. 63.16 on Flickr30K variant)

Fine-tuning stage (LLaVA-v1.5-7B, 5K SiTe-ratio vs. baseline): - Spatial-MM: +4.19% (46.21 vs. 42.02) - MMEPosition: +13.17 points (141.00 vs. 127.83)

Fine-tuning stage (Qwen2-1.5B, 5K SiTe vs. baseline): - MMBench: +5.33% (67.00 vs. 61.67) - MM-VetSpat: +4.38% (64.25 vs. 60.00)

3D/real-world generalization (LLaVA-v1.5-7B, Flickr30K+SiTe-ratio): - CV-Bench 3D: +11.41% (39.80 vs. 28.39) - RealWorldQA: +1.74% (56.25 vs. 54.51)

HALVASiTe outperforms HALVABaseline on most benchmarks using only spatial token swaps as negatives, vs. HALVA's full object/relation/attribute modifications.

Baselines: standard pretraining (no augmentation), Rotate, Crop, VSR data substitution, MixGen (implicitly). SiTe outperforms all on spatial benchmarks consistently.

Limitations

  • Small capacity models (Qwen2-1.5B) show slight COCO-QA regression during fine-tuning, attributed to over-adaptation to binary Yes/No QA format reducing open-ended generalization.
  • VQA-v2 and MM-Vet show slight drops in supervised fine-tuning for LLaVA, suggesting domain shift from high-spatial-supervision instruction data.
  • Noun extraction via Qwen2.5-72B during fine-tuning introduces a dependency on a large external model (though caption-text extraction is used at pretraining scale).
  • Stitched images may suffer from uneven scaling or blank regions under random pairing (SiTe-rand); SiTe-ratio partially mitigates this but adds a filtering/grouping step.
  • Spatial relations generated are limited to left/right and top/bottom from stitching axes; more complex 3D or occlusion-based spatial relations are not directly injected.
  • Evaluation confined to 2D stitching; depth/3D spatial grounding is tested only as zero-shot generalization.

Relevance to Vision-Language Models

SiTe directly addresses a well-documented failure mode of VLMs—spatial hallucination—by targeting the training data distribution rather than the model architecture or decoding strategy, making it universally applicable to existing pipelines. It contributes a scalable recipe for improving cross-modal alignment on spatial semantics, a capability increasingly critical for downstream VLM applications in embodied AI, navigation, and robotics. The work is relevant to researchers studying data-centric methods for VLMs, as it demonstrates that structured compositional augmentation of image–text pairs can provide supervision signal as effective as costly annotation or generative rewriting. The contrastive hard-negative extension connects to the broader literature on hallucination mitigation (e.g., HALVA, VCD) and offers a simple alternative construction mechanism.