Skip to content

Bridging the gap to real-world language-grounded visual concept learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a scalable framework for language-grounded visual concept learning in real-world scenes, overcoming the limitations of prior work (LIVCL) that required manually predefined concept axes and per-axis encoders. A pretrained VLM discovers concept axes adaptively per image, a single Q-Former-based universal concept encoder binds visual features to those axes, and a compositional anchoring loss enforces axis disentanglement without sacrificing instance-specific detail.

Problem

Existing language-grounded visual concept learning (e.g., LIVCL) is restricted to a small set of manually predefined primitive axes (color, shape, style), relies on per-concept specialized encoders that scale poorly, and grounds concepts by directly aligning concept embeddings to image-agnostic text embeddings—forcing a trade-off between disentanglement and instance-specific detail. These limitations confine the approach to synthetic, low-complexity datasets.

Method

Adaptive Concept Axes Discovery: A pretrained VLM (InternVL2-5) is prompted with a universal prompt containing a general task description and a single output exemplar. The VLM returns a per-image dictionary mapping concept axis names to textual descriptions (e.g., {'age': 'young', 'gender': 'male'}), all in one query. The number of axes K varies per image.

Universal Concept Encoder (UCE): A Q-Former architecture replaces its standard learnable queries with text embeddings of the discovered axis names. Cross-attention between these axis-conditioned queries and DINOv2 visual features binds each axis to its corresponding visual concept representation, all within a single parameter set—no per-axis encoder needed.

Compositional Anchoring Loss (L_Comp): During training, concept representations from the same axis are randomly swapped across images within a batch, producing composite representations Z^c. The frozen T2I decoder (Stable Diffusion at 256×256, 10 DDIM steps) renders composite images x^c = D(Z^c). A lightweight shared regression MLP g_φ, built on top of a fixed copy of the encoder re-encoding x^c, predicts textual descriptions; cosine distance to the swapped text embeddings provides the disentanglement signal. Crucially, z_i is never forced to equal the text embedding v_i, preserving instance-level visual nuance. The total loss is L_Diff + λ_Comp · L_Comp + λ_Reg · L_Reg.

Key Contributions

  • Adaptive per-image concept axis discovery via a single-query universal VLM prompt, requiring no prior knowledge of dataset attributes.
  • A unified Q-Former-based concept encoder that handles arbitrarily many concept axes without introducing per-axis parameters.
  • Compositional anchoring objective that achieves axis disentanglement in generated image space rather than by collapsing to text embeddings, retaining instance-specific visual details.
  • Demonstrated generalization to open-vocabulary, never-seen-in-prompt concept axes (e.g., breed, nose color, eye color) and out-of-distribution concept compositions.

Results

  • Visual concept editing (single axis, Table 1): Outperforms LIVCL and four text-based baselines (SDEdit, InstructPix2Pix, NullText Inversion, DDPM-Inversion) on CLIP-Score and BLIP-Score across all four datasets (ImageNet-S20, CelebA-HQ, AFHQ-Dog, AFHQ-Cat). Example: CelebA-HQ CLIP 0.239 vs. LIVCL 0.226; BLIP 0.496 vs. LIVCL 0.469.
  • Human evaluation (Table 2): Larger margin over baselines; Ours: CelebA-HQ 0.636, AFHQ-Dog 0.589, AFHQ-Cat 0.623 vs. best baseline DDPM-Inversion at 0.528/0.584/0.548.
  • Multi-image composition (Table 3, N=2,3,4 axes): Consistently best; baselines degrade with more axes whereas the proposed method shows only marginal drops (e.g., CelebA-HQ BLIP: 0.492/0.490/0.491 vs. LIVCL 0.454/0.440/0.429).
  • Ablations (Table 4–6): UCE vs. shared MLP: CLIP 23.88 vs. 21.63. DINOv2 vs. CLIP encoder: CLIP 23.88 vs. 22.03. Both L_Comp and g_φ are individually necessary; removing either degrades CLIP/BLIP by ~1–3 points.
  • Robust to VLM choice (Qwen2.5-VL, Ovis2, InternVL2-5 all comparable) and to 10–20% random axis drop.

Limitations

  • Evaluated only at 256×256 resolution; scalability to higher resolution is not demonstrated.
  • Depends on VLM output quality for axis discovery; while shown robust to partial drops, very poor VLM coverage of relevant axes could harm performance.
  • Compositional anchoring requires candidate images sharing the same discovered axis within each training batch; rare axes may not find suitable swap partners.
  • No quantitative evaluation of disentanglement beyond the editing proxy metric (CLIP/BLIP against swapped descriptions).
  • Inference requires an additional VLM query per image; runtime cost relative to baselines is not reported.

Relevance to Vision-Language Models

This work directly leverages a pretrained VLM (InternVL2-5) as a zero-shot concept axis discoverer, making VLM grounding capability the backbone of a downstream representation learning pipeline—a paradigm increasingly central to VLM research. It demonstrates that a single universal prompt can transfer VLM semantic understanding to continuous visual representation learning without task-specific fine-tuning. For researchers tracking VLMs, the paper contributes evidence that VLM-guided prompting can replace manually engineered label spaces in structured visual learning, and that image-space (rather than embedding-space) alignment objectives better preserve the rich multimodal features VLMs have learned. It complements work on compositional generalization in VLMs and the use of VLMs as labelers or structure-discoverers in downstream vision tasks.