Skip to content

RAVEN: An Agentic Framework for Multimodal Entity Discovery from Large-Scale Video Collections

🕒 Published (v1): 2025-03-03 08:28 UTC · Source: Arxiv · Venue: AAAI 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RAVEN is a two-stage agentic framework that autonomously discovers and extracts structured, domain-specific entities from large-scale video collections by first inferring canonical categories and generating schemas, then performing schema-guided multimodal entity extraction. It operates across visual, audio, and textual modalities using pluggable VLMs and LLMs. Evaluated on 1.5 million clips, it substantially outperforms unimodal baselines in entity recall.

Problem

Existing VLM/LLM-based video understanding methods process videos in isolation and lack mechanisms for collection-wide, structured entity extraction. There is no prior system that can infer domain-specific schemas from unlabeled video corpora and apply them consistently across millions of clips without requiring manual schema authoring or domain expertise upfront.

Method

RAVEN operates in two sequential agentic stages:

  1. Category Understanding & Schema Generation: A VLM (Gemini 1.5 Flash) processes each video clip to infer raw categories and generic entities (people, objects, locations). An LLM (GPT-4o) normalizes and deduplicates raw category labels into a canonical list, then generates per-category entity schemas—typed entity names plus typed attributes with example values—indexed for retrieval.

  2. Rich Domain-Specific Entity Extraction: Each clip is re-processed by the VLM. The canonicalized category is used to retrieve the most semantically similar schema (via embedding similarity on raw vs. canonical category names). The schema and synthetic examples are injected into the VLM prompt via in-context learning, producing structured JSON entities aligned to the schema. Results are persisted for downstream indexing.

Model-agnosticism is enforced by requiring only that the backend VLM/LLM support comparable context lengths and structured JSON output.

Key Contributions

  • Two-stage agentic pipeline separating category canonicalization from entity extraction, requiring no prior domain knowledge of the collection
  • Dynamic schema generation: LLM derives entity types and typed attributes per inferred domain, eliminating manual schema authoring
  • Schema-guided, example-conditioned prompting for in-context extraction of fine-grained attributes and relationships (e.g., Event → Date, Person → Role)
  • Model-agnostic design: any VLM/LLM pair satisfying context-length and structured-output constraints can be substituted

Results

  • Scale: Category understanding run over 1.5 million video clips (>5,000 hours) from the Aligned Video Captions dataset; entity richness evaluation on a 300-clip stratified sample
  • Entity recall (Person, Location, Object vs. baselines on 300 clips):
  • RAVEN achieves the highest recall across all three generic entity types
  • NER-on-Speech, OCR, visual caption keyword extraction, and YOLOv10 each fail to recover entities not present in their respective modalities (e.g., YOLO produces no Event or Location structure; OCR misses non-textual scene entities)
  • Qualitative depth (Abraham Lincoln documentary case study): RAVEN extracts Historical Event → Description/Date/Location/Key Figures and Historical Site → Era/Architectural Features; baselines return only flat labels ("Person", "Train") or isolated name strings with no attributes
  • No quantitative precision/F1 metrics are reported; evaluation is recall-based and qualitative

Limitations

  • No quantitative precision or F1 evaluation; no human-annotated ground truth is described, making it difficult to assess false-positive rates
  • Schema generation quality is not ablated; errors in LLM-generated schemas propagate silently into extraction
  • Semantic retrieval for schema lookup (raw → canonical category) is not benchmarked; mismatch could cause systematic schema misapplication
  • Evaluation sample (300 clips) is small relative to the 1.5M-clip corpus; diversity of failure modes is underexplored
  • Entity relationship extraction (e.g., cross-entity links) is deferred to future work
  • Computational cost of double VLM passes per video is not analyzed

Relevance to Harnesses / Meta-Harnesses

RAVEN is a textbook agentic meta-harness: it orchestrates multiple specialized model calls (VLM for perception, LLM for normalization and schema synthesis, VLM again for schema-conditioned extraction) with structured inter-stage data flow and dynamic prompt generation, all without human-in-the-loop intervention. The schema generation step is itself a harness primitive—an LLM is used to produce the prompting scaffolding consumed by a downstream agent stage, directly paralleling meta-harness patterns where one agent generates the configuration or tool specification for subsequent agents. The model-agnostic, modular design (swap any VLM/LLM satisfying interface contracts) reflects the plug-in harness architecture common in pipeline orchestration frameworks. For researchers tracking harnesses, RAVEN illustrates how schema-as-code generated by LLMs can serve as a dynamic control structure replacing hand-authored prompt templates.