StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description¶

🕒 Published (v1): 2026-07-13 16:50 UTC · Source: Arxiv · Venue: ECCV · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

StoryTeller is a training-free framework for long-form audio description (AD) that maintains a persistent narrative state—an identity graph and a salience-weighted fact memory—across video clips, enabling later descriptions to reference characters and events established in earlier scenes. It requires only raw video and a movie title, with no subtitles, character banks, face-identity databases, or task-specific fine-tuning. The authors also introduce StoryAD-QA, a narrative-comprehension QA benchmark that tests whether a language model can answer story-context questions using only the generated AD text.

Problem¶

Modern VLMs are effective on short clips but treat each clip independently, producing AD that omits character identity, causal relationships, and cross-scene story continuity needed by blind and low-vision (BLV) audiences. Existing training-free AD systems (e.g., AutoAD-Zero, MM_Narrator) still depend on precomputed character banks, aligned subtitles, or AD transcripts, and propagate context only in shallow static forms (e.g., concatenated previous descriptions) without modeling which narrative facts should remain salient and which should fade. Standard metrics (CIDEr, BLEU, SPICE) evaluate lexical overlap against reference captions, not whether a listener can reconstruct the story from the AD.

Method¶

StoryTeller processes clips $\{v_1,\ldots,v_T\}$ chronologically, maintaining a narrative state $\mathcal{S}_t = (\mathcal{G}_t, \mathcal{M}_t)$ updated by $\mathcal{S}_t = \Phi(v_t, \mathcal{S}_{t-1})$.

Identity Graph $\mathcal{G}_t$: Faces are detected per frame and grouped into temporal tracklets; a pretrained ArcFace encoder $\phi$ produces per-tracklet embeddings $\mathbf{e}_k = \frac{1}{|\mathcal{T}_k|}\sum_{i\in\mathcal{T}_k}\mathbf{v}_{t,i}$. Cosine similarity against existing identity means $\boldsymbol{\mu}_j$ determines assignment ($j^* = \arg\max_j \cos(\mathbf{e}_k,\boldsymbol{\mu}_j)$; create a new node if below threshold $\tau$). Characters begin anonymous and are named only when VLM verification accepts the association.

Grounded Fact Induction: A VLM proposes structured candidates $f = (\text{subject}, \text{action}, \text{object}, \text{context})$, optionally conditioned on IMDb plot/character metadata retrieved by embedding similarity to the current scene summary $\mathbf{s}_t$. Candidates pass a semantic filter: $g_f = \max(\cos(\mathbf{z}_f, \mathbf{s}_t),\, \max_m \cos(\mathbf{z}_f, \varepsilon(f_m))) > \sigma$, then a dedicated VLM verifier issues ACCEPT/REJECT against the video clip. Metadata can only suggest hypotheses; no fact enters memory unless visually verified.

Salience-Based Narrative Memory $\mathcal{M}_t$: Each verified fact $f_m$ carries a salience weight updated via reinforcement–decay: $$w_m^{(t)} = \lambda\, w_m^{(t-1)} + \alpha\, r_m^{(t)}, \quad r_m^{(t)} = \cos(\mathbf{s}_t, \mathbf{z}_m)$$ where $\lambda\in(0,1)$ is a decay factor and $\alpha$ is reinforcement strength. Narratively persistent facts accumulate weight; transient details decay and fade.

Key Contributions¶

Training-free long-form AD from raw video + movie title only, with no subtitles, scripts, character banks, face identity databases, or task-specific fine-tuning
Dynamic identity graph that progressively resolves anonymous tracklets into named characters through accumulating visual and narrative evidence
Salience-based narrative memory with reinforcement–decay dynamics modeling human-like story salience across scenes
StoryAD-QA: a 2,574-question multiple-choice benchmark (Track A: 1,611 segment-only; Track B: 963 context-conditioned) measuring whether generated AD supports downstream narrative reasoning, assessed without access to the video

Results¶

StoryTeller consistently improves narrative coherence, factual grounding, and story comprehension over strong baselines (including AutoAD-Zero) across automatic, QA-based, and human evaluations on standard AD benchmarks (LSMDC, MAD-Eval) and diverse long-form videos
StoryAD-QA contains 2,574 questions derived from MAD-Eval; Track B questions require cross-scene context to answer correctly, directly probing narrative preservation
Specific accuracy scores and numerical deltas over baselines are reported in evaluation sections not included in the provided text excerpt

Limitations¶

Identity assignment depends on ArcFace cosine similarity with a fixed threshold $\tau$; performance degrades with poor face detection (occlusion, low resolution, non-frontal angles)
VLM verification may still accept incorrect facts if metadata and visual evidence are both ambiguous; the system cannot retract facts already written into earlier AD clips
IMDb metadata availability is title-dependent; films with sparse public metadata receive fewer character-name hints
Sequential, clip-by-clip processing precludes retroactive correction of early descriptions when a character's identity is resolved later
QA questions in StoryAD-QA are automatically generated by a VLM, introducing potential errors or biases in the benchmark itself

Relevance to Vision-Language Models¶

StoryTeller demonstrates a concrete method for extending VLMs beyond their effective context window for long-form video understanding by externalizing narrative state into a structured, dynamically maintained memory rather than relying on in-context concatenation. The semantic filtering and VLM-verification loop directly addresses the hallucination problem in retrieval-augmented VLM pipelines, showing how to gate external knowledge (metadata) through visual grounding. The StoryAD-QA benchmark provides a VLM-agnostic protocol for measuring story comprehension—a capability dimension orthogonal to the lexical metrics dominant in video captioning evaluation. For researchers tracking VLMs, this work establishes a reproducible, resource-light baseline for identity-consistent long-form video narration that can be adopted with any underlying VLM backbone.