Skip to content

GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memory

🕒 Published (v1): 2025-11-15 04:29 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GCAgent is an agent-based framework for long-video understanding that constructs structured episodic memory—encoding schematic (event roles/templates) and narrative (causal/temporal ordering) structures—from speech transcripts before any query arrives. A two-agent system (Memory Manager + Reasoning Agent) then runs a Perception–Action–Reflection cycle to answer queries grounded in both global context and retrieved local evidence. On Video-MME Long split, it achieves 73.4% accuracy, a 23.5% improvement over the vanilla Qwen2.5-VL backbone.

Problem

MLLMs handling long videos face two compounding failures: (1) token-window limits prevent ingesting entire videos, and (2) existing agent-based retrieval methods are purely query-reactive—they retrieve local snippets but never build a coherent global understanding, missing long-term causal/temporal dependencies that humans maintain via schematic and narrative cognition.

Method

Memory Construction (pre-query): Speech transcripts (from VAD + ASR) are fed to a GPT-4.1-Mini Memory Manager Agent. A structured prompt segments transcripts into coherent episodes, each tagged with a title, description, keywords, causal_link (inter-episode causality), and narrative_role (one of: Introduction, Development, Turning Point, Conflict, Climax, Resolution, Epilogue). This produces a JSON-serialized episodic memory \(M = \{m_1, \ldots, m_L\}\) capped at 100 episodes. When audio is unavailable, uniform-sampled frames are captioned instead.

Query-time Reasoning (three stages): 1. Perception: Memory Manager retrieves query-relevant transcript spans and maps them to video segments: \(V^*, T^* = A_\text{perc}(Q, O, T, M)\). 2. Action: Reasoning Agent (Qwen2.5-VL 7B, ≤32 frames) jointly reasons over \((Q, O, V^*, T^*, M)\) to produce answer \(a^*\) and visual evidence \(E^*\): \(a^*, E^* = A_\text{act}(Q, O, V^*, T^*, M)\). 3. Reflection: Memory Manager integrates \(E^*\) back into \(M\) to yield updated memory \(M' = A_\text{refl}(Q, O, a^*, E^*, M)\), enriching context for subsequent queries.

Key Contributions

  • Global-Context-Aware Agent Framework: First agent framework that constructs full episodic memory before query arrival, bridging global context modeling with query-conditioned retrieval.
  • Schematic + Narrative Episodic Memory: Explicit JSON schema encoding event-level abstractions (schematic) and causal/temporal inter-event relations (narrative), grounded in cognitive psychology models.
  • Perception–Action–Reflection Cycle: Structured three-stage query-time pipeline that grounds answers in both global episodic context and locally retrieved multimodal evidence.
  • Audio-Unavailable Fallback: Visual captioning path for videos lacking speech, preserving framework applicability across domains.

Results

  • Video-MME Long split: 73.4% accuracy (+23.5% over vanilla Qwen2.5-VL 7B baseline without agent).
  • Video-MME Overall Average: 71.9% — highest among comparable 7B-scale MLLMs.
  • Video-MME Short/Medium/Long: 72.6% / 69.8% / 73.4%.
  • LongVideoBench: 55.0%.
  • Outperforms agentic 7B baselines VideoTree, VideoRAG, and DrVideo on Video-MME.
  • Approaches proprietary model performance (GPT-4o, Gemini-1.5-Pro achieve >70% average on both benchmarks).

Limitations

  • Memory construction relies on speech transcripts; ASR/VAD errors propagate directly into episodic memory quality.
  • Visual-only fallback (image captioning) is less semantically rich than transcript-based memory, and the paper does not benchmark this path separately.
  • Memory is capped at 100 episodes—very long or densely-eventful videos may require coarser granularity.
  • Reasoning agent is limited to 32 frames, which may miss fine-grained visual evidence not captured by temporal boundaries from transcripts.
  • GPT-4.1-Mini used as Memory Manager introduces a closed-source API dependency and adds latency/cost relative to single-model baselines.

Relevance to Harnesses / Meta-Harnesses

GCAgent is a concrete instantiation of a multi-agent meta-harness over a base MLLM: it wraps Qwen2.5-VL with an orchestrating Memory Manager Agent that decomposes the long-video task into pre-query memory construction and a query-time Perception–Action–Reflection sub-pipeline, exactly the pattern of a harness coordinating specialist sub-agents with structured state. The episodic memory acts as the harness's shared state store—analogous to a context accumulator in a digest harness—updated incrementally via the Reflection stage after each reasoning call. The explicit JSON schema for inter-agent communication (episode entries with typed fields) is a reusable design pattern for harnesses where multiple agents must read and write shared structured context without ambiguity.