Skip to content

VideoAgent: All-in-One Framework for Video Understanding and Editing

🕒 Published (v1): 2026-06-22 13:37 UTC · Source: Arxiv · link

Why this paper was selected

All-in-one video understanding/editing agent; long-horizon multi-task orchestration architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoAgent is an agentic framework that unifies video understanding and editing through a shot planning pipeline and a multi-agent graph orchestration system with over 30 specialized editing agents. It introduces textual-gradient graph optimization—iterative LLM-driven refinement of DAG-structured agent workflows—to handle diverse, long-form video creation tasks from natural language instructions. It achieves 87–95% workflow orchestration success rates and reduces API costs by 60% versus baselines.

Problem

Existing automated video systems are limited to short-segment processing and domain-specific tasks; none support coherent long-form narrative creation or dynamic orchestration across heterogeneous multimodal editing tools. The two core gaps are: (1) lack of global-aware planning for multi-shot, long-duration video, and (2) no adaptive mechanism to compose and validate complex, multi-agent workflows for arbitrary editing instructions.

Method

VideoAgent operates in two main stages:

Stage 1 — Automated Video Shot Creation: 1. A Shot Planning Agent compresses captions of all source material keyframes into a visual summary \(C\), then generates an ordered shot storyboard \(S = (s_1, \ldots, s_{|V|})\) satisfying the user instruction \(I\). 2. A Cross-Modal Retrieval module indexes keyframes via ImageBind/CLIP embeddings and retrieves the best-matching clip per shot by cosine similarity: \(r_i = \arg\max_j \cos(e_j, E(s_i))\). 3. A VLM-based Trimming Agent selects precise sub-segment timestamps \([t_{\text{start}_i}, t_{\text{end}_i}]\) from the retrieved clip to match target shot duration and narrative.

Stage 2 — Multi-Agent Graph Orchestration: Given an instruction, an Intent Parsing mechanism extracts required intents \(T\). The system then searches for an optimal DAG \(G^* = (A, E)\) over the agent library \(A\) (30+ specialized agents) by minimizing: $\(G^* = \arg\min_{G \in \mathcal{G}} \left[ \mathcal{L}_{\text{struct}}(G) + \lambda \mathcal{L}_{\text{align}}(G, I) \right]\)$ where \(\mathcal{L}_{\text{struct}}\) penalizes cycles and disconnected components, and \(\mathcal{L}_{\text{align}}\) penalizes intent coverage gaps. Optimization is performed via Textual-Gradient Descent: at each iteration \(t\), a quality signal \(Q^{(t)} = (\tau, \kappa, \chi)^{(t)}\) (acyclicity, intent coverage, edge compatibility) is computed, and an LLM generates natural-language "gradients" that modify graph topology: $\(G^{(t+1)} = G^{(t)} \otimes \nabla^{\text{text}}_G \mathcal{L}(G^{(t)})\)$ Optimization terminates when both losses vanish or iteration count reaches \(T_{\max}\).

Key Contributions

  • Unified agentic framework covering both video understanding (retrieval) and editing (production) from a single natural language instruction.
  • Global-aware shot planning agent that generates coherent multi-shot storyboards by reasoning over compressed representations of all source materials.
  • Textual-gradient graph optimization for iterative DAG-structured agent workflow construction, replacing single-pass LLM graph generation with an explicit structural+semantic optimization loop.
  • VideoEdit benchmark: a new evaluation suite with high-quality instructions and human-aligned LLM-judge evaluation across six video genres.
  • 60% reduction in API costs via intent-filtered tool selection that avoids feeding all candidate material into context.

Results

  • Workflow orchestration success rate: 87–95% across four backbone LLMs (Claude-Sonnet-3.7/4, GPT-4o, Deepseek-v3), surpassing best baseline by 2–25%.
  • Video retrieval (Shot2Story): Recall@1 improves from 31.22% → 48.85% (GPT-4o), 30.04% → 44.93% (Gemini-2.5-flash), and 46.03% → 44.27% (Claude-Sonnet-3.7, slight drop in Recall but IoU improves 23.91% → 24.81%).
  • API cost: 60% reduction vs. baselines (e.g., cost drops from $0.374 to $0.147 per query with Claude-Sonnet-3.7).
  • Human evaluation: VideoAgent rated only 4% below human-created professional videos across 18 trending videos; outperforms NoteBookLM, Director, FunClip, NarratoAI across all 6 video categories.
  • Ablation: Removing gradient optimization (-G) drops success rate from >90% to below 55%; removing shot planner causes significant retrieval degradation.
  • Scaling: More optimization rounds consistently improve success rate and reduce performance gaps between backbone LLMs.

Limitations

  • Evaluation is on a newly proposed benchmark (VideoEdit) introduced by the same authors, with limited independent validation.
  • The textual-gradient optimization adds iterative LLM calls; latency and total cost at inference time for complex workflows are not fully characterized.
  • Convergence to \(\mathcal{L}_{\text{struct}} = 0 \wedge \mathcal{L}_{\text{align}} = 0\) is not guaranteed; a hard iteration cap \(T_{\max}\) is the fallback.
  • Agent library of 30+ tools is manually curated; extensibility and robustness to new tool integration are not systematically evaluated.
  • Human evaluation uses 26 participants on 18 videos—sample size limits statistical power for fine-grained quality comparisons.

Relevance to Harnesses / Meta-Harnesses

VideoAgent is a concrete production-oriented meta-harness: it wraps over 30 heterogeneous task-specific agents under a single orchestration layer that dynamically selects, composes, and validates their execution graph from a natural language intent. The textual-gradient graph optimization is a principled, iterative refinement mechanism for DAG-structured agent pipelines—directly relevant to harness research on how to auto-assemble and validate multi-agent workflows without hand-coded routing. The explicit separation of intent parsing, structural validity, and semantic alignment as distinct optimization objectives offers a reusable template for meta-harness designers building general-purpose agent orchestrators. The 60% API cost reduction from targeted intent filtering demonstrates that harness-level routing decisions have measurable efficiency impact beyond correctness.