Object-centric binding in Contrastive Language-Image Pretraining

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OC-CLIP augments CLIP-style contrastive pretraining with object-centric inductive biases: a slot-attention binding module that maps scene-graph nodes to image-patch slots, and a structured similarity score that replaces CLIP's single-vector cosine similarity. This directly targets the binding problem—CLIP's failure to correctly associate attributes and spatial relationships with the right objects in multi-object scenes—without relying on hard-negative augmentation.

Problem

CLIP learns bag-of-words-style representations via a single global vector, causing it to fail at compositional scenes: it cannot reliably bind attributes to objects (e.g., "red square and blue circle" vs. "blue square and red circle") or understand spatial relationships (near-random performance on COCO-spatial and GQA-spatial even after in-domain finetuning). Existing fixes add hard-negative text/image augmentations, which improve surface-level sensitivity but plateau and do not generalize to unseen compositions because they don't address the representational root cause.

Method

OC-CLIP inserts two new components between the frozen/pretrained encoder backbones and the contrastive loss:

1. Binding module: An inverted cross-attention layer where queries are text-entity embeddings (from a scene graph parsed by an LLM, e.g., Llama3) and keys/values are image patch embeddings from an intermediate ViT layer. Softmax is applied along the query dimension, forcing slots to compete for patches. A set of learned "default query tokens" absorbs uncaptioned visual content to stabilize training; these tokens are discarded afterward.

2. Structured similarity score: Replaces the single cosine similarity with a weighted sum of (a) per-node object scoring—cosine similarity between each textual node embedding and its assigned visual slot—and (b) relationship scoring—a learned MLP function of the relation embedding and the subject/object slot embeddings. Weights α and β are learned.

Training losses: (1) Image-text contrastive loss using the structured score; (2) relationship loss that penalizes symmetric relationship scores by contrasting against scene graphs with swapped subject/object and randomly shuffled relations.

The vision and text backbones (OpenCLIP or DINOv2) are initialized from pretrained weights; only the binding module is trained from scratch.

Key Contributions

• Introduction of OC-CLIP: a CLIP pretraining framework with object-centric binding inductive biases via slot attention and scene graphs.
• Binding module using inverted cross-attention with default tokens to cleanly separate visual entities corresponding to scene-graph nodes.
• Structured similarity score that decomposes image-text alignment into per-object and per-relation terms.
• Demonstration of sample efficiency over hard-negative augmentation strategies in a controlled synthetic environment (PUG-based 3D dataset).
• Scalability results when trained from scratch on CC3M+CC12M (15M noisy image-text pairs).

Results

Attribute binding (SugarCrepe & ARO-A): - OC-CLIP (B-14) vs. OpenCLIP-FT: +16.5% on swap-attribute, +20.4% on swap-object, +4.1% on replace-relation. - OC-CLIP (B-14) vs. best large-scale hard-negative/dense-captioning baseline (DAC): +13.6% on swap-attribute, +8.4% on swap-object.

Spatial relationship understanding (Whatsup & ARO-R): - OC-CLIP (B-14) vs. OpenCLIP-FT: +44.1% on COCO-spatial (89.7% vs. 45.6%), +43.6% on GQA-spatial (92.7% vs. 49.1%), +34.8% on ARO-R (84.9% vs. 50.1%). - Outperforms XVLM (best prior contrastive baseline) on all spatial benchmarks by 10–27%.

Synthetic binding (PUG 3D): - OC-CLIP with 30% training pairs and 0% hard-negatives reaches 90–93% accuracy vs. OpenCLIP plateauing at ~80% with 70% pairs and 70% hard-negatives (seen & unseen pairs).

Scaling from scratch (CC3M + CC12M = 15M): - +12.8% zero-shot ImageNet classification vs. OpenCLIP trained identically. - +12.7% SugarCrepe swap-attribute, +6.6% swap-object in zero-shot setting.

Limitations

• Requires a scene-graph parser at both training and inference time; LLM-based parsers introduce their own biases and failure modes.
• OC-CLIP's new alignment space is built around slot embeddings, dropping the CLS token; generalization is bounded by the training vocabulary (cannot trivially inherit OpenCLIP's broad zero-shot coverage).
• Evaluated only at ViT-B scale and up to 15M training images; has not been scaled to 400M+ images needed to match production CLIP variants.
• Computational overhead from the binding module and structured similarity not fully analyzed in the main text (deferred to Appendix A.3).
• Relationship scoring relies on LLM-extracted scene graphs, which may be noisy or absent for images with implicit spatial relationships.

Relevance to Vision-Language Models

This paper directly attacks the compositional understanding gap of CLIP-family models, a long-standing weakness that has resisted hard-negative augmentation approaches. It bridges object-centric representation learning (slot attention) with contrastive VLM pretraining—a connection largely unexplored prior to this work—and shows that structural inductive biases are more sample-efficient than data augmentation for binding. The structured similarity score is a reusable architectural primitive that could be layered on top of any patch-based vision encoder (they test both OpenCLIP and DINOv2), making this relevant to practitioners fine-tuning or building on CLIP for downstream tasks requiring spatial or attribute reasoning. The work sets a strong baseline for compositional benchmarks (SugarCrepe, ARO, Whatsup) and motivates further exploration of scene-graph-conditioned alignment in larger-scale VLM pretraining pipelines.