Skip to content

SAGE: A Unified Framework for Generalizable Object State Recognition with State-Action Graph Embedding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SAGE is a unified framework for recognizing physical object states in instructional videos by decomposing states into language-described visual concepts organized in a State-Action Graph, constructed via LLMs and refined with VLMs. Unlike prior specialist models, SAGE generalizes to unseen objects and actions without retraining. It achieves up to 14.6% relative precision improvement over the prior SOTA on novel objects/actions while using less than 5% of its inference time.

Problem

Pretrained VLMs fail at fine-grained physical object state recognition (e.g., "whole lemon" vs. "sliced lemon") because their training objectives neglect state transformations. Existing specialized methods (LFC, VidOSC) train per-action/object classifiers with fixed heads, making generalization to unseen objects or actions impossible by design. No prior work offers a unified, open-world model for object state recognition that covers both seen and novel action–object combinations.

Method

SAGE constructs a State-Action Graph where each action node (verb–object pair) connects to three state nodes (initial, transitioning, end), each described by a set of fine-grained visual concepts generated by an LLM (GPT-4o-mini). State embeddings are computed as the mean of their visual concept text embeddings (via CLIP's text encoder); action embeddings are the vector difference between end and initial state embeddings, following translational relation encoding. The graph is refined by ranking concepts on their cosine similarity to actual video frames and enforcing that ≥50% of selected concepts are shared across multiple object states (top-5 per state, ≥3 shared). A temporal Transformer processes per-frame CLIP visual features; pseudo frame-level labels are generated by VLM cosine similarity and constrained via Viterbi decoding (enforcing initial→transitioning→end order). At inference, the action is predicted from video-level embeddings against SAGE action embeddings; object states are then decoded with Viterbi over the action-specific state set.

Key Contributions

  • State-Action Graph Embedding (SAGE) that represents object states as averages of shared visual concept embeddings, enabling compositional generalization to unseen objects and actions.
  • Multimodal graph refinement procedure that selects visually reliable and cross-state-shared concepts using VLM frame–concept similarity scores.
  • Viterbi-constrained pseudo-label generation for learning from unlabeled instructional videos with temporal ordering enforcement.
  • A unified single model that subsumes specialist baselines and adds open-world generalization without requiring action or object labels at inference time.
  • A new evaluation protocol for the fully novel setting (both action and object unseen during training).

Results

  • ChangeIt, novel objects + novel actions (State Pre@1): SAGE 0.45 vs. VidOSC 0.27 and LFC 0.21 — +66.7% relative over VidOSC.
  • HowToChange, novel objects + novel actions (Pre@1): SAGE 45% vs. VidOSC 37% and LFC 27%.
  • Known objects (ChangeIt, State Pre@1), unified setting: SAGE 0.51 vs. MTC 0.47, VidOSC* 0.41.
  • Inference runtime (unknown actions/objects): SAGE 29.7 s vs. VidOSC 612.7 s and LFC 7278.4 s on HowToChange — <5% of VidOSC's time.
  • With VideoCLIP backbone (Table 7): SAGE outperforms VidOSC on ChangeIt State Pre@1 (0.60 vs. 0.57 seen; 0.55 vs. 0.48 novel) and HowToChange Pre@1 (63.6% vs. 60.7% seen; 61.2% vs. 58.2% novel).
  • Graph refinement improves novel Pre@1 on HowToChange from 45% (before) to 50% (after) with the same 5-concept budget.

Limitations

  • Evaluated only on single-action-per-video benchmarks (ChangeIt, HowToChange); SAGE cannot directly handle concurrent multi-object state transformations without an upstream temporal action localization module.
  • Requires an LLM at graph construction time (GPT-4o-mini or Qwen3-32B), adding a dependency outside the main training loop.
  • The model is purely a state/action recognizer; integration into a general vision foundation model is left as future work.
  • Pseudo-label quality depends on the frozen VLM used for cosine similarity estimation, which is a known noise source.

Relevance to Vision-Language Models

SAGE directly addresses a documented failure mode of pretrained VLMs — their inability to recognize fine-grained physical object states — and demonstrates how LLM-generated visual concept decompositions can be grafted onto CLIP-style embeddings to recover this capability. The framework treats VLMs as a source of structured multimodal knowledge (concept scoring, pseudo-label generation) rather than as end-to-end classifiers, a pattern increasingly relevant for compositional and open-world VLM applications. Results are robust across CLIP, SigLIP, MetaCLIP, and VideoCLIP backbones, suggesting SAGE is a plug-in improvement rather than a backbone-specific trick. For researchers working on VLM grounding, temporal video understanding, or robotic perception, SAGE provides a concrete technique for bridging language-described visual attributes with video-level physical state dynamics.