Co-Director: Agentic Generative Video Storytelling

🕒 Published (v1): 2026-04-27 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Co-Director is a hierarchical multi-agent framework that formalizes generative video storytelling as a global optimization problem, coupling a Multi-Armed Bandit (MAB) over a factored creative action space with a local MLLM self-refinement loop. It addresses cascading failures and semantic drift endemic to linear "waterfall" agentic pipelines by centrally parameterizing creative intent and propagating it top-down to all sub-agents. On GenAd-Bench (400 advertising scenarios), it achieves an average score of 81.4 versus 65.3 for the best prior agentic baseline.

Problem

Existing agentic video pipelines chain LLM modules in fixed "waterfall" sequences: early prompt errors propagate and amplify downstream (cascading failures), individual sub-agents interpret creative intent independently causing semantic drift, and static handcrafted templates cannot explore the non-linear creative latent space. The classical credit assignment problem makes it difficult to trace terminal failures to specific upstream decisions.

Method

Co-Director consists of four hierarchically nested agents: an Orchestrator (\(\Phi_{orch}\)), Pre-Production (\(\Phi_{pre}\)), Production (\(\Phi_{prod}\), itself containing Keyframe, Video, and Audio sub-agents), and Post-Production (\(\Phi_{post}\)).

Global optimization (MAB): The pipeline is non-differentiable, so Co-Director treats it as a black box steered by a centralized creative configuration \(\theta = (\theta_{cs}, \theta_{nm}, \theta_{aa})\) across three orthogonal axes: Creative Strategy (Informational / Transformational / Comparative), Narrative Mode (Analytical / Vignette / Drama), and Aesthetic Archetype (4 options). At each of \(T\) iterations, UCB1 selects an arm combination; the Orchestrator synthesizes \(\theta^{(t)}\) into agent-specific instructions injected into all sub-agent system prompts, ensuring coherent top-down conditioning. A factored reward \(R^{(t)} = (r_{cs}, r_{nm}, r_{aa}) \in [0,100]^3\) from an MLLM judge updates each bandit arm independently, analytically bypassing credit assignment. An LLM-driven warm start initializes arm priors from product/demographic context before the first pipeline execution.

Local optimization (self-refinement): Before storyboard expansion, an LLM scores the storyline \(L\) on hook quality, cohesion, and prompt adherence; if below threshold \(\tau_L\), it regenerates \(L\) using revised prompts. Similarly, an MLLM evaluates the keyframe sequence \(K\) jointly for sequence-level consistency; flagged keyframes are selectively regenerated while accepted ones are preserved.

Key Contributions

• Hierarchical multi-agent architecture (\(\Phi_{orch} \to \Phi_{pre} \to \Phi_{prod} \to \Phi_{post}\)) treating video storytelling as a global optimization problem rather than a linear chain.
• Factored MAB over a three-axis creative action space \(\theta = (\theta_{cs}, \theta_{nm}, \theta_{aa})\) with UCB1 and LLM warm-start, enabling principled exploration-exploitation of the creative latent space.
• Two-level optimization: global MAB for creative configuration search; local MLLM self-refinement loops on storyline and keyframes.
• GenAd-Bench: 400 fictional-brand advertising scenarios (200 products × 2 demographics) with MLLM-as-a-Judge scoring on VAF, DA, MA, and VQ, validated against human raters.
• Open-source release of Co-Director and GenAd-Bench.

Results

• Co-Director (\(T=4\)): 81.4 avg vs. best prior agentic baseline MovieAgent: 65.3 and AniMaker: 62.2 (using identical underlying models for a controlled comparison).
• Base Agentic Pipeline (\(T=1\), same orchestration, no iterative refinement): 68.5 — proving core orchestration alone outperforms prior linear pipelines.
• Random Search (\(T=4\), no MAB/warm-start): 75.7 — MAB adds +5.7 avg over Best-of-N random sampling.
• VQ (Visual Quality): Co-Director 70.2 vs. raw Veo 3.1 backbone 50.5, demonstrating optimization recovers quality beyond single-pass generation.
• Warm-Up + MAB converges to a high-quality plateau earlier than Random Search; Random Search eventually closes the gap only at much higher iteration budgets (Figure 2 extrapolation to \(T=10\)).
• MLLM judge alignment approaches human-human agreement ceiling (validated in human study, Section 5.2).

Limitations

• Evaluated only on video advertising; generalization to broader cinematic narratives is claimed but not quantitatively demonstrated in the paper.
• \(T=4\) pipeline iterations with full video generation per iteration is computationally expensive; cost/efficiency trade-offs are deferred to Appendix F.
• MAB operates over a small discrete action space (3 × 3 × 4 = 36 arms); expressiveness is bounded by this hand-designed taxonomy grounded in marketing theory.
• MLLM-as-a-Judge evaluation, while validated against human raters, may carry model-specific biases (all evaluation uses Gemini 3 Pro, same family as the generator).
• Final videos are fixed at 4 shots / 12 seconds, limiting evaluation of long-horizon storytelling.

Relevance to Harnesses / Meta-Harnesses

Co-Director is a canonical example of a meta-harness: a controller layer that does not itself generate content but orchestrates, steers, and globally optimizes a pipeline of specialized generative sub-agents. The MAB-over-pipeline pattern — treating a chained multi-agent workflow as a black-box arm to be scored and re-sampled — is directly applicable to any non-differentiable harness where end-to-end gradient flow is unavailable. The hierarchical parameterization strategy (centralized configuration \(\theta\) injected top-down into sub-agent system prompts) offers a concrete mechanism for maintaining semantic coherence across independently-prompted sub-agents, a problem that recurs in all multi-step harness designs. The warm-start technique (LLM-initialized prior before first execution) is a practical pattern for reducing cold-start cost in iterative harness loops.