Soap2Soap: Long Cinematic Video Remaking via Multi-Agent Collaboration

🕒 Published (v1): 2026-05-17 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Soap2Soap is a multi-agent framework for series-level cinematic video remaking—replacing actors or styles across hundreds of shots while preserving narrative, motion, and identity. It enforces long-horizon consistency through a Dual-Bridge mechanism (structured JSON screenplay + visual reference anchors) and a closed-loop verification agent that triggers selective shot regeneration. On the SoapBench benchmark, it outperforms commercial APIs (Kling O1, Runway Gen4) and open-source baselines across identity, scene, and plot consistency metrics.

Problem

Existing video generation pipelines degrade over long temporal horizons (hundreds of shots) due to compounding identity drift, background mutation, and semantic erosion—problems that physical continuity naturally solves in real filmmaking but must be explicitly engineered in generative remaking. No prior system addresses series-level (episode/film-scale) video-to-video transformation with joint enforcement of character identity, narrative causality, and motion choreography.

Method

Soap2Soap orchestrates three specialized agents under a shared contextual framework called Dual-Bridge Consistency:

1. Language Bridge (\(S_{json}\)): A scene-aware JSON screenplay produced by a long-context video understanding model (Gemini 3 Flash). It encodes shot-level narrative events, camera language, character participation, and per-shot T2I/I2V generation prompts—serving as the persistent semantic backbone.

2. Visual Bridge (\(M\)): An external visual memory storing scene-level environment anchors and shot-level character reference images. For shot \(s_{i,j}\), a compact memory package \(M_{i,j} = \text{Allocate}(S_{json}, s_{i,j})\) is dynamically constructed, retrieving only the minimal context needed for that shot to avoid role confusion from global context overload.

3. Batch Keyframe Consistency: Multiple keyframes sharing a scene are jointly generated as a \(2 \times 2\) or \(3 \times 3\) grid in a single diffusion pass (via Nano Banana 2), allowing intra-grid cross-attention to suppress identity drift under viewpoint changes. Grids are then split into individual keyframes.

4. Video Synthesis: Google Veo 3 performs image-to-video synthesis per shot (4–8 second clips), conditioned on the keyframe and \(M_{i,j}\), then clips are stitched temporally.

5. Verification Agent (closed-loop): Gemini 3 Flash audits each generated keyframe and clip against \((S_{json}, M_{i,j})\) across four dimensions—generation quality, identity/appearance, environmental stability, and plot consistency. On failure, it produces structured feedback \(\Delta\) and triggers: $\(\bigl(M'_{i,j},\, P'_{\text{I2V}}\bigr) = \text{Refine}(M_{i,j},\, S_{json},\, \Delta)\)$ with selective per-shot regeneration rather than full-sequence rollback.

Key Contributions

• Formal definition of long-video cinematic remaking as a video-to-video generation task spanning hundreds of shots with joint consistency constraints.
• Dual-Bridge Consistency: decouples semantic structure (JSON screenplay) from visual appearance (visual memory anchors), making long-range consistency an explicit, controllable objective rather than an emergent property.
• Batch keyframe grid synthesis: joint \(2 \times 2\)/\(3 \times 3\) grid generation in a shared latent context for intra-scene consistency under large viewpoint changes.
• SoapBench: a new benchmark with 10 movies, 607 annotated shots, covering both long-video understanding (character/plot accuracy vs. IMDb scripts) and remaking evaluation (identity, scene, and plot consistency).
• Training-free, heterogeneous multi-agent implementation using state-of-the-art foundation models.

Results

• On the long-video remaking track of SoapBench (VLM scores on 10-point scale; CLIP-I scores):

Method	ID-VLM	Scene-VLM	Plot-VLM	CLIP-I (ID)	CLIP-I (Scene)
Mocha	6.21	6.74	5.52	0.563	0.681
Kling O1	8.11	8.37	8.60	0.632	0.751
Runway Gen4	7.48	7.96	8.27	0.604	0.724
Soap2Soap	9.17	8.84	8.67	0.842	0.819

• Soap2Soap achieves the largest margin in CLIP-I (ID): +0.210 over Kling O1, indicating substantially stronger character identity preservation.
• Strongest gains over open-source baseline Mocha: +2.96 ID-VLM, +2.10 Scene-VLM, +3.15 Plot-VLM.
• Human study results referenced but not fully reported in the provided text.

Limitations

• Evaluation scale is limited: remaking track uses only 10 movie segments of 1.5–5 minutes each (up to 42 consecutive shots), which may not capture full-episode drift.
• VLM-as-a-judge evaluation uses Gemini 3 Flash—the same model family used for understanding and verification—creating potential evaluator bias (partially mitigated by model-agnostic CLIP scores, but not eliminated).
• Computational cost of selective regeneration loops is configurable but not formally analyzed; retry budget must be manually tuned.
• Relies on proprietary, closed APIs (Veo 3, Nano Banana 2, Gemini 3 Flash)—no open-source replication path.
• SeedDance 2.0 (a potentially strong baseline) excluded due to API unavailability and portrait restrictions, leaving the comparison incomplete.

Relevance to Harnesses / Meta-Harnesses

Soap2Soap is a concrete implementation of a production-grade multi-agent harness where the "meta" layer—the Dual-Bridge Consistency mechanism—explicitly manages state (JSON screenplay + visual memory) shared across subordinate agents, rather than relying on emergent coordination. The closed-loop Verification Agent embodies the plan-execute-verify-refine loop central to meta-harness design: it audits outputs, generates structured corrective feedback \(\Delta\), and routes it back to upstream agents for selective repair—a reusable pattern applicable beyond video to any long-horizon sequential generation task. The contextual memory allocation module (\(M_{i,j} = \text{Allocate}(S_{json}, s_{i,j})\)) addresses a core meta-harness challenge: dynamic scoping of context to avoid overloading individual agents with irrelevant global state. For researchers building harnesses over generative pipelines, Soap2Soap demonstrates that separating semantic coordination (language bridge) from visual coordination (visual bridge) into orthogonal state stores significantly improves end-to-end consistency.