Causal Graphical Models for Vision-Language Compositional Understanding

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

COGT (Causally-Ordered Generative Training) replaces standard autoregressive or parallel caption decoding in VLMs with a partially-ordered, semi-parallel prediction strategy governed by a Causal Graphical Model (CGM) built from dependency parse trees. By conditioning each token prediction only on its syntactic ancestors plus visual features, the model avoids spurious word-order correlations that cause bag-of-words failures in contrastive VLMs. Trained only on COCO (~100K samples), COGT sets a new state of the art across five compositional benchmarks by a large margin.

Problem

CLIP-based VLMs treat captions as bags of words—unable to distinguish "the horse is eating the grass" from "the grass is eating the horse"—because the contrastive loss collapses text to a single vector and standard autoregressive (AR) training introduces spurious inter-token correlations tied to English word order rather than true syntactic/semantic dependency structure.

Method

A frozen VLM visual encoder (CLIP, XVLM, or InstructBLIP) provides visual tokens Z drawn from the last two encoder layers. An off-the-shelf dependency parser (Deep Biaffine + RoBERTa; Dozat & Manning, 2016) converts a caption into a Dependency Tree (DT). The DT is then reinterpreted as a CGM: each word variable Wⱼ has parents {all transitive ancestor words Wᵢ₁…Wᵢₖ, syntactic-type variable Sⱼ, all visual variables Z₁…Zₘ}. The joint distribution is factorized as the product of these per-node conditionals (disentangled CGM factorization), discarding all inter-token dependencies not licensed by the parser.

A small 3-block decoder D (~64M parameters) implements this: each block contains (1) a Dependency Guided Attention layer where masked token embeddings mⱼ attend only to their parent visible tokens vᵢ₁…vᵢₖ (no cross-masked attention), and (2) a cross-attention layer over visual features Z. There are 45 syntactic-category-specific initial masked token embeddings (one per Universal Dependencies tag), conditioning the decoder on Sⱼ. Inference scores candidate captions by level-order traversal of the DT, predicting all tokens at depth d in parallel before moving to depth d+1.

Key Contributions

• CGM-guided semi-parallel AR prediction for compositional VLM adaptation (COGT framework)
• Dependency Guided Attention: each masked token attends only to syntactically licensed parent visible tokens, not all prior words
• 45 syntactic-type-specific masked token embeddings as discrete syntactic conditioning
• Multi-layer visual features (last + penultimate CLIP encoder layers) to preserve spatial/small-object information
• Introduction of FG-OVD (Fine-Grained Open-Vocabulary Detection) as a new compositional benchmark focusing on fine-grained object attribute discrimination
• Demonstrated generality across three VLM backbones (CLIP, XVLM, InstructBLIP) without modifying the visual encoder

Results

• COGT-CLIP vs. all COCO-only baselines: achieves average 75.20 across five benchmarks vs. second best DAC-LLM (trained on CC3M, 33× larger) at 57.86 — a +17.34-point gap
• COGT-CLIP vs. second best in Tab. 3 overall (DAC-LLM): +12.27 points average
• COGT-CLIP+ vs. best prior method: ~22 points higher average across all benchmarks
• Ablation — prediction strategy comparison (all on COCO, CLIP backbone):
• Fully-Parallel: 57.42 avg; Mixed: 67.10; Sequential-AR: 69.28; COGT: 75.20
• COGT vs. Fully-Parallel: +17.77 pts; vs. Sequential-AR: +5.92 pts
• Ablation — component removal: dropping syntactic-type-specific masks: −2.69 pts avg; using only last CLIP layer: −4.75 pts avg
• COGT-XVLM (COCO only): outperforms CE-XVLM on all evaluated datasets; avg 77.10 vs. N/A for CE
• COGT-InstructBLIP (COCO only): avg 77.87, surpassing Cap/CapPa (pre-trained on private data orders of magnitude larger)
• Downstream classification (Tab. 6): COGT-CLIP improves CIFAR-100 from 79.0→84.9 and ImageNet top-1 implicitly via stronger representations

Limitations

• Requires an external dependency parser at both train and test time; parser errors directly degrade the CGM structure
• The sparseness assumption (DT as causally sufficient) may miss true long-range semantic dependencies not captured by syntax
• COGT cannot be applied to tasks requiring full autoregressive generation (it scores candidates, not generates freely)
• Better parsers would likely improve results but publicly available high-performing ones are limited
• FG-OVD benchmark was tested only on cropped object bounding boxes, limiting scope of the compositional evaluation
• Winoground performance is not reported in the main tables (addressed in appendix), as the benchmark has known noise issues for CLIP-based methods

Relevance to Vision-Language Models

This paper directly targets the bag-of-words failure mode endemic to CLIP-family VLMs, offering a principled structural fix via causal modeling rather than data augmentation (hard negatives) or scale. The CGM factorization is backbone-agnostic and lightweight (~64M parameter decoder), making it a practical adapter for existing frozen VLM encoders. It reframes the compositionality problem as one of spurious correlation removal — a framing with implications for grounding, relational reasoning, and retrieval in multimodal systems. The result that a small causally-structured decoder trained on 100K COCO samples outperforms methods trained on 3.3M noisy web captions suggests CGM-based inductive biases may be a more data-efficient path to compositional VLM improvement than scaling data.