How can embedding models bind concepts?¶

🕒 Published (v1): 2026-05-29 16:23 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

Seong Joon Oh (Tübingen); ICML 2026; concept binding failure in CLIP-style models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CLIP and similar dual-encoder vision-language models fail at cross-modal concept binding (e.g., distinguishing "red square, blue circle" from "blue square, red circle") not because object information is absent, but because CLIP learns a high-complexity binding function that cannot generalize to unseen concept combinations. The authors show scene embeddings decompose additively into object components, which explains why uni-modal probes succeed, and demonstrate that binding generalization is achievable in controlled transformer models that instead learn low-complexity multiplicative binding functions.

Problem¶

Vision-language embedding models behave as bag-of-concepts systems: they recognize individual concepts (red, square) but fail at cross-modal object recognition—identifying which concept values co-occur to form a specific object in a multi-object scene. Prior work observed that CLIP has uni-modal binding (recoverable via probes) but fails cross-modally, without explaining the geometric mechanism behind this asymmetry or whether the failure is fundamental.

Method¶

The authors formalize binding through two criteria: concept recognition (ranking present concept values above absent ones via cosine similarity) and object recognition (ranking present objects, i.e., concept combinations, above absent ones). Binding requires satisfying both cross-modally.

Geometric analysis of CLIP: They estimate object embeddings $u_o$ by averaging encoder outputs over scenes containing object $o$ (variants: AVG, AVG+POS, SINGLE-OBJ) and test whether scene embeddings satisfy a two-level additive decomposition: $$f(x_s) \approx u_{o_1} + u_{o_2} \approx \sum_{i=1}^C u_{c_{1,i}} + \sum_{i=1}^C u_{c_{2,i}}$$ They validate this via $R^2$, retrieval accuracy, and probing accuracy, and further test it through object-replacement interventions: $\tilde{z} = f(x_s) - u_{o_1} + u_{o'_1}$.

Binding function complexity: They train a 1-hidden-layer ReLU MLP $g: \mathcal{C}^2 \rightarrow \mathbb{R}^d$ to approximate the binding map from discrete concept indices to CLIP scene embeddings, varying MLP width $\{64, 256, 1024, 4096\}$ and training coverage fraction $\{10\%, \ldots, 90\%\}$. Generalization is measured on held-out objects.

Controlled models: Transformer-based dual-encoders are trained from scratch on synthetic multi-object data with controlled training coverage to test whether binding generalization emerges, and to characterize the structure of generalizing binding functions (additive vs. multiplicative interactions).

Key Contributions¶

Establishes that CLIP scene embeddings admit a two-level additive decomposition: scenes → objects → concepts, with object components directly responsible for uni-modal binding
Demonstrates that object-level components support meaningful counterfactual edits in embedding space via linear arithmetic
Shows that CLIP's binding function is high-complexity: MLP approximators with large capacity and high training coverage still fail to generalize to unseen concept combinations
Shows that binding generalization is not fundamental: controlled transformer models achieve it with sufficient data coverage
Identifies multiplicative concept interactions as the structural signature of generalizing binding functions, in contrast to CLIP's effectively additive composition

Results¶

CLIP text encoder: object-based scene reconstruction $R^2 = 0.90$ (AVG) / $0.92$ (AVG+POS); retrieval accuracy $0.97$; probing accuracy $0.99$
CLIP image encoder (PUG:SPARE): $R^2 = 0.75$ / $0.84$; retrieval $0.93$; probing $0.98$
Object replacement intervention (CLIP, CLEVR): retrieval $1.00$, probing $0.98$–$0.99$ with AVG+POS embeddings
Removing concept components from CLIP text embeddings collapses concept decoding: $1.00 \rightarrow 0.06$, while object decoding is largely preserved ($1.00 \rightarrow 0.99$)
Removing object components collapses both object and concept decoding: text $0.04$ / $0.05$, image $0.01$ / $0.02$
CLIP binding function approximation (MLP): object recognition on held-out objects remains low regardless of MLP width or training data fraction, confirming high complexity

Limitations¶

Geometric analysis is primarily on synthetic or controlled datasets (CLEVR, CLEVR-2D, PUG:SPARE) and CLIP/DINOv2; generalization to other VLM architectures (e.g., contrastive models with richer text encoders) is not demonstrated
The two-object scene focus may not scale cleanly to real-world images with many overlapping or occluded objects, though 3-object CLEVR and Gemini-generated natural images are briefly examined
The controlled models that achieve binding generalization are trained from scratch on synthetic data; it is unclear how to transfer these insights to fine-tuning or re-training CLIP at scale
The paper is truncated before reporting quantitative results from controlled model experiments

Relevance to Vision-Language Models¶

This work provides a mechanistic explanation for CLIP's compositional binding failures, a long-studied limitation of VLMs in tasks requiring attribute-object binding (e.g., Winoground, ARO). By localizing the failure to the complexity of the binding function rather than the absence of object structure, it redirects research from data augmentation or architecture changes toward the geometry and algebraic structure of the embedding space. The finding that multiplicative concept interactions are necessary for generalizable binding has direct implications for designing next-generation contrastive or dual-encoder models. It also clarifies why fine-tuning approaches that add word-order or attribute-association losses have limited success: the binding function geometry must change, not just the training signal.