Skip to content

Crayotter: Traceable Multi-Agent Workflows for Long-Form Video Editing

🕒 Published (v1): 2026-05-31 14:07 UTC · Source: Arxiv · link

Why this paper was selected

Traceable multi-agent workflows preserving narrative evidence across video editing pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Crayotter is an open-source multi-agent system that decomposes long-form video editing into three auditable phases—material preparation, editing research, and tool-grounded execution—each externalizing explicit artifacts (coverage reports, blueprints, tool logs, renders) to enable traceable, selectively repairable production runs. Against two baselines on 23 themes, it achieves a human evaluation score of 3.40/5 vs. 2.44 (CapCut-Mate) and 1.70 (CutClaw). The system also defines a replayable trajectory schema and RLVR interface for future policy optimization.

Problem

Long-form video editing requires coordinating heterogeneous decisions (asset retrieval, timeline construction, narration alignment, post-production) over a long horizon, but existing systems treat the process as a black box: no inspectable intermediate state, no replay capability, no localized failure diagnosis. A single error in clip timing or asset selection can propagate undetected through the entire production.

Method

Crayotter factorizes the transformation \(F: (q, A_0) \to y\) into three sequential phases with an explicit artifact contract:

  1. Phase 1 – Coverage-Aware Retrieval: The request \(q\) is expanded into weighted semantic tags \(T = T^{\text{scene}} \cup T^{\text{act}} \cup T^{\text{style}} \cup T^{\text{story}} \cup T^{\text{shot}}\). A coarse retriever scores candidates by \(\alpha \cdot s_{\text{text}} + (1-\alpha) \cdot s_{\text{meta}}\); per-candidate tag support \(g(v, t_i)\) is estimated via a multimodal module over sampled frames and temporal windows. Reranking combines request relevance with marginal coverage gain \(D_i(v, S)\). The loop iterates until normalized coverage \(\text{Cov}(S, T) \geq \eta\) or a retrieval budget is exhausted.

  2. Phase 2 – Editing Blueprint: Time-marked views \(\tilde{v}_j = R_{\text{time}}(v_j)\) are synthesized without executing any editing tools. A research agent localizes narrative beats \(\{z_m\}\) to source intervals \((\hat{j}_m, \hat{\tau}_m)\) via timestamp-watermarked frames, producing a temporally addressable blueprint \(B = \{b_m\}\) encoding clip, interval, cinematic role, transition rationale, and narration intent.

  3. Phase 3 – ReAct Execution: The blueprint is executed as an artifact transition MDP: \(s_{k+1} = E(s_k, a_k; B)\). After each tool call, a verifier emits diagnostics \(\zeta_k\) (tool validity, timestamp accuracy, coverage preservation, render quality). Failed diagnostics trigger localized repair of only the affected segment. A 21-tool taxonomy covers search, ranking, download, analysis, cutting, merging, transitions, continuity scoring, timeline assembly, narration, audio post-processing, subtitles, and export.

The RLVR interface models Phase 3 as an episodic MDP \(\mathcal{M}^{\text{RL}}\) and optimizes a policy \(\pi_\theta\) via GRPO with a three-tier decomposed reward: \(r_k = \lambda_{\text{exec}} r_k^{\text{exec}} + \lambda_{\text{art}} r_k^{\text{art}} + \lambda_{\text{align}} r_k^{\text{align}} + \lambda_{\text{ref}} r_k^{\text{ref}}\), plus an episode-level judge reward and optional human-preference calibration.

Key Contributions

  • Traceable artifact formulation: source pool \(P = (S, \Gamma, M)\), blueprint \(B\), and tool history \(H = \{(s_k, a_k, s_{k+1}, \zeta_k)\}\) as explicit inspectable representations.
  • Coverage-aware multimodal retrieval loop with tag decomposition, marginal-coverage reranking, and iterative gap-filling.
  • Three-phase open-source editing system evaluated on 23 themes with released traces exposing intermediate decisions.
  • Replayable trajectory schema and three-tier RLVR reward design (verifiable tool signals + outcome + human preference) ready for policy optimization with GRPO.

Results

  • Human evaluation (23 themes, 3 annotators): Crayotter 3.40/5 vs. CapCut-Mate 2.44 and CutClaw 1.70.
  • Consistent gains across all five dimensions: theme alignment, content richness, narrative coherence, editing smoothness, visual quality.
  • Both AI and human scoring reported; human scores aggregated at the case level to avoid annotator weighting bias.

Limitations

  • Evaluation covers only 23 themes, limiting generalizability; no large-scale automatic benchmark specific to editing-quality axes (cut continuity, transition naturalness) is used.
  • The RLVR interface is described but not yet demonstrated with trained policies; the paper presents the design, not empirical RL results.
  • Retrieval depends on external video platform APIs and TTS/ASR services, introducing latency and availability constraints not analyzed in the paper.
  • Phase 2 blueprint accuracy depends on frozen MLLMs for temporal localization; temporal-watermarking is a workaround rather than a model-level solution.
  • No ablation over individual phases or reward components is reported to isolate their contribution.

Relevance to Harnesses / Meta-Harnesses

Crayotter is a concrete realization of a production-oriented agent harness with an explicit artifact contract, phase router, runtime watchdog, bounded experience memory, and deterministic replay capability—precisely the structural features that distinguish a harness from an ad-hoc LLM pipeline. Its three-tier RLVR reward design and replayable trajectory schema form a meta-harness layer: the harness both executes production runs and generates the supervision signal needed to optimize the agent policy that runs inside it. The pattern of externalizing every intermediate state as an inspectable artifact (rather than keeping it in latent dialogue) directly parallels process-oriented deep-research harnesses (e.g., DeepPresenter, SAGE) and is a transferable design principle for any long-horizon harness that needs failure attribution and selective re-execution. Practitioners building harnesses for other domains will find the coverage-aware retrieval loop, artifact-contract specification, and the decomposed verifiable-reward cascade directly applicable.