Skip to content

Open Ad-hoc Categorization with Contextualized Feature Learning

🕒 Published (v1): 2025-12-18 05:49 UTC · Source: Arxiv · Venue: CVPR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OAK introduces open ad-hoc categorization, a task where a model must infer latent organizational contexts (e.g., Action vs. Location vs. Mood) from a few labeled exemplars and discover novel categories in unlabeled data. It achieves this by prepending small learnable context tokens to a frozen CLIP ViT, jointly optimized with CLIP's image-text alignment and GCD's visual clustering objectives. On Stanford and Clevr-4 benchmarks, OAK achieves 70% Omni accuracy vs. 52% for GCD and 43% for CLIP+LLM-vocab.

Problem

Standard visual recognition assumes a fixed, globally shared semantic space—a single correct interpretation per image. Real AI agents must organize the same visual scene differently depending on their current goal (e.g., a household robot alternating between recognizing Actions for assistance, Locations for navigation, and Moods for emotional responsiveness). Existing methods—open-vocabulary classifiers like CLIP and generalized category discovery (GCD) methods—each address only one dimension: semantic breadth or visual clustering, but not context-adaptive switching between multiple latent categorization principles.

Method

OAK introduces a small set of learnable context tokens z_c prepended to the patch tokens of a frozen CLIP ViT-B/16 encoder, producing a context-specific encoder f_c(x_i) = f([x_i, z_c]). One independent set of tokens is trained per latent context (Action, Location, Mood, etc.). The training objective combines:

  1. GCD loss (bottom-up): self-supervised + supervised contrastive losses over visual embeddings, enabling novel visual cluster discovery.
  2. Text guidance loss (top-down): cross-entropy against CLIP text embeddings for both labeled data (ground-truth labels) and unlabeled data (pseudo-labels from semi-supervised K-means with Hungarian matching at each epoch), with novel class names generated by prompting an LLM.

Full objective: ℓ_OAK(z_c) = ℓ_GCD(z_c) + λ_text · ℓ_text(z_c). Context switching at inference requires only swapping z_c; the backbone is never modified.

Key Contributions

  • Formalizes open ad-hoc categorization: joint inference of latent context and expansion of both known and novel categories (semantic + visual) from few-shot labeled exemplars.
  • Proposes Omni accuracy: a metric requiring correct prediction across all contexts simultaneously for the same image, testing representation consistency under context switching.
  • Develops OAK: a minimal, frozen-backbone model using context tokens to achieve context-aware attention without any direct supervision on the tokens themselves.
  • Demonstrates interpretable context-conditioned saliency maps (hands for Action, faces for Mood, backgrounds for Location) without explicit saliency supervision.

Results

  • Stanford dataset (novel accuracy): OAK 87.4% on Mood vs. CLIP-ZS 35.4% and GCD 40.6% — >50% absolute gain.
  • Stanford Omni accuracy (overall): OAK 70.3% vs. GCD 52.3% and CLIP-ZS + LLM vocab 43.0%.
  • Clevr-4 few-shot Omni (novel): OAK 16.5% vs. GCD 15.7%; OAK 40.5% Omni overall vs. GCD 22.6%.
  • Clevr-4 full-shot (novel accuracy): OAK 83.3% average vs. µGCD 78.0%; OAK gains +11% on Texture (66.5% vs. 55.5%).
  • OAK achieves near-perfect Color (100%) and Shape (99.9%) accuracy on Clevr-4 in both few-shot and full-shot settings.

Limitations

  • Omni accuracy evaluation on Stanford is unreliable due to only 8/17 labeled/novel overlapping images across contexts—results are de-emphasized in the paper.
  • Assumes a known number of novel classes per context (shared with GCD baseline).
  • Context names are never provided to the model; novel class names rely on LLM-generated vocabulary, which can inherit LLM biases.
  • Evaluated only on static images; the paper notes extension to video/dynamic settings as future work.
  • Few-shot setup may underperform compared to prior full-shot GCD works since only 16 labeled images per class are used.

Relevance to Agentic AI / LLM Agents

This work directly targets a capability gap for embodied and task-driven AI agents: the ability to reorganize perception according to the current goal without retraining. The Omni accuracy metric formalizes context switching—a household robot must simultaneously maintain Action, Location, and Mood representations and switch between them seamlessly—which is a concrete desideratum for multi-task agents. OAK's design (frozen backbone + swappable lightweight tokens) is a tractable architectural pattern for agents that must handle many ad-hoc tasks without catastrophic forgetting. The integration of LLM-generated vocabularies for novel class naming also demonstrates a practical human-in-the-loop pattern where language models expand an agent's categorical vocabulary from minimal supervision.