Skip to content

Show Me Examples: Inferring Visual Concepts from Image Sets

🕒 Published (v1): 2026-07-02 16:35 UTC · Source: Arxiv · link

Why this paper was selected

Apple Research (Susskind/Bautista); fills fundamental VLM gap: concept inference from image sets without text

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VICIS (Visual Concept Inference from Sets) is a new task that evaluates whether a model can infer a shared visual concept from a small set of context images and apply it to a query image to generate outputs consistent with that concept — all without language or labels. State-of-the-art VLMs fail this task, typically ignoring visual context and defaulting to biased or query-duplicating generations. The authors propose a purpose-built architecture (Set Learner + Instantiation Module + Diffusion Model) trained via weak supervision from the WordNet/ImageNet hierarchy that substantially outperforms both general-purpose VLMs and task-specific baselines.

Problem

Current VLMs exhibit strong instruction-following from text but lack robust visual in-context learning: given only a set of images sharing an implicit concept, they cannot reliably infer that concept and transfer it to a new query image. Models either ignore the context set, reproduce a dominant query concept, or copy the query — indicating they treat example images as noise rather than as nonverbal concept specifications.

Method

The framework has three components trained end-to-end with a rectified flow (flow matching) objective:

  1. Set Learner: A Vision Transformer (ViT) that receives the concatenated token embeddings of all context images (each encoded by a pretrained ViT encoder \(E\)) and predicts \(k\) direction vectors \(\mathbf{D}_c = \{\mathbf{d}_1, \ldots, \mathbf{d}_k\}\) spanning the concept-specific subspace of the feature space.

  2. Instantiation Module: Projects the query image's CLS-token embedding \(\mathbf{e}^{\text{query}}\) onto each concept direction via dot products, \(s_i = \langle \mathbf{e}^{\text{query}}, \mathbf{d}_i \rangle\), then reconstructs a concept-conditioned token that discards irrelevant query information: $\(\mathbf{e}^{\text{query}}_{\text{proj}} = \sum_{i=1}^{k} s_i \mathbf{d}_i\)$

  3. Diffusion Model: Conditioned on \(\mathbf{e}^{\text{query}}_{\text{proj}}\) injected into the timestep embedding, trained with the flow matching loss \(\mathcal{L}_{\text{FM}}\) to denoise target images consistent with the inferred concept instantiation.

Training data is constructed without per-concept annotation by leveraging the WordNet hierarchy over ImageNet: context sets are assembled from sibling nodes at one hierarchy level, and query–target pairs share the same child node, providing scalable weak supervision.

Key Contributions

  • Introduces VICIS: a nonverbal visual in-context learning task requiring concept inference from image sets and concept-guided generation, without text, labels, or fine-tuning.
  • Demonstrates that state-of-the-art open-source and proprietary VLMs (ILLUME+ 3B, BAGEL 7B-MoT) largely fail VICIS, exposing a concrete gap in current visual reasoning.
  • Proposes a scalable training framework using WordNet/ImageNet hierarchical groupings as weak supervision for episode construction.
  • Introduces a dual metric of accuracy (classifier-verified hierarchy correctness) and diversity (entropy-ratio penalizing query copying) to jointly evaluate concept inference quality.
  • Demonstrates generalization to out-of-domain modalities (ImageNet Sketch) and unseen classes (ImageNet-21k).

Results

All results from Table 1; "per con." = accuracy averaged over concepts, "per inst." = averaged over concept instantiations:

  • Hierarchy dataset — Ours: 46.34% / 54.46% / 0.81 diversity; ILLUME+ 3B: 37.15% / 55.00% / 0.57; BAGEL 7B-MoT: 26.05% / 33.76% / 0.46; Visual Prompting: 26.02% / 45.89% / 0.70.
  • Sketch queries — Ours: 32.60% / 47.11% / 0.72 vs. Visual Prompting: 23.76% / 44.57% / 0.68.
  • Sketch context — Ours: 41.50% / 52.24% / 0.79 vs. Visual Prompting: 27.67% / 47.04% / 0.71.
  • Sketch context + query — Ours: 31.90% / 46.47% / 0.71 vs. Visual Prompting: 25.69% / 44.45% / 0.68.
  • ImageNet-21k unseen classes — Ours: 39.63% / 51.40% / 0.79 vs. Visual Prompting: 29.07% / 49.95% / 0.76.
  • Ours dominates the accuracy–diversity Pareto front across all conditions.

Limitations

  • Quantitative evaluation is restricted to the animal subtree of the WordNet/ImageNet hierarchy, which may not represent all visual concept types.
  • Training still requires auxiliary hierarchical groupings (WordNet); fully unsupervised concept discovery is not addressed.
  • The paper text is truncated, so stated limitations in later sections (discussion, conclusion) are not accessible.
  • Comparison against proprietary models (Table 4, referenced but absent in provided text) is only partially reported.
  • The model is task-specific and requires training for VICIS; it is not a plug-in improvement to existing VLMs.

Relevance to Vision-Language Models

This work directly probes and quantifies a blind spot in modern VLMs: the ability to perform visual in-context learning without language scaffolding. By constructing a rigorous benchmark that separates genuine concept inference from bias-driven generation, the paper provides a diagnostic tool for the VLM community. The finding that large, general-purpose models (BAGEL 7B, ILLUME+ 3B, and reportedly leading proprietary models) systematically fail VICIS suggests that scaling text-image training alone is insufficient for visual reasoning, motivating architectures with dedicated set-reasoning inductive biases. The proposed method also offers a blueprint for injecting concept-conditioned control into diffusion-based VLMs without per-concept fine-tuning.