Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper formalizes Multi-Scale Temporal Prediction (MSTP) — jointly forecasting scene states across multiple temporal horizons and state granularities — and proposes IG-MC, a closed-loop VLM-based framework combining incremental diffusion-based visual generation with a decision-driven multi-agent collaboration system. A new benchmark (MSTP Benchmark) covering general and surgical scenes with synchronized multi-scale annotations is also introduced. IG-MC is a plug-and-play module that boosts Qwen2.5-VL accuracy by up to +46 points over the base VLM.

Problem

Existing VLMs handle single-scale temporal prediction (fixed temporal windows or single state granularity), failing to model the hierarchical and time-varying nature of workflow decision-making. Long-horizon prediction compounds errors because visual token accumulation degrades performance, and there is no benchmark providing synchronized multi-scale (both temporal and state) annotations for evaluating unified MSTP.

Method

IG-MC comprises two plug-and-play modules layered on top of an existing VLM (Qwen2.5-VL-7B-Instruct in experiments):

Incremental Generation (IG): At each incremental timestep τ, a Decision-Making module (DM) predicts the next state Sₖ₊₁ from the current state Sₖ and visual guidance Iₖ. A Visual Generation module (VG) — a modified Stable Diffusion with tripartite conditioning (state embedding via cross-attention, prior image via residual injection, latent projection for domain plausibility) — then synthesizes the next visual preview Iₖ₊₁ conditioned on Sₖ₊₁ and Iₖ. This alternating DM→VG→DM loop uses only the current state-image pair (stateless), avoiding temporal buffer accumulation.

Decision-Driven Multi-agent Collaboration (MC): A State Transition Controller (STC) VLM-agent receives Sₖ and Iₖ and decides whether a state transition is needed and at which hierarchical level l. If triggered, a chain of L LLM-based State Prediction agents (v₁…vₗ) sequentially refine predictions from coarse phase to fine step; coarser levels (i < l) carry forward unchanged. The STC is fine-tuned with a data augmentation strategy that rebalances the severe temporal sparsity of state transitions (~100:1 ratio) to 1:1 by sampling synthetic transitions around real ones.

The DM and VG are trained with separate objectives and combined at inference (plug-and-play into any base VLM).

Key Contributions

• MSTP task formalization: decomposes multi-scale into two orthogonal axes — temporal scale (look-ahead intervals) and state scale (phase→step hierarchy) — in both general and surgical scenes.
• MSTP Benchmark: first dataset with synchronized multi-scale annotations; MSTP-General (Action Genome + 3 state scales × 3–60s temporal scales) and MSTP-Surgery (GraSP + 2 state scales × 1/5/30/60s).
• Incremental Generation: stateless DM+VG feedback loop that prevents error accumulation over long horizons without growing temporal buffers.
• Multi-agent Collaboration: hierarchical STC + L-level prediction agents with transition-sparsity augmentation for balanced fine-tuning.
• Plug-and-play compatibility: IG-MC modules attach to any base VLM without architectural modification.

Results

All improvements are relative to Qwen2.5-VL-7B-Instruct baseline (no IG-MC):

• MSTP-Surgery, 60s temporal scale: Accuracy 13.20% → 51.90% (+38.70pp) with DM; F1 3.83% → 22.65% (+18.82pp) with DM+VG.
• MSTP-Surgery, 5s temporal scale: Accuracy 12.00% → 50.80% (+38.80pp) with DM.
• MSTP-Surgery, 30s temporal scale: Recall 4.02% → 33.99% (+29.97pp) with DM+VG (incremental 5s).
• MSTP-General, 10–20s temporal scale: F1 10.24% → 51.07% (+40.83pp) with DM+VG.
• MSTP-General, 20–30s temporal scale: F1 3.88% → 59.36% (+55.48pp) with DM+VG.
• Temporal stability: at 5s temporal scale, Accuracy degrades only 3.62% from time step 1 to 5, confirming long-horizon robustness.
• Visual generation quality degrades gradually with horizon (FID increases from ~80 to ~100, SSIM decreases) but remains functional for downstream state prediction grounding.

Limitations

• VG module can hurt accuracy when incremental scale is finer than temporal scale (e.g., 1s incremental for 5s temporal on Surgery: Accuracy drops 8.4pp from DM-only to DM+VG), suggesting generated visual quality introduces noise at high-frequency synthesis.
• Visual prediction metrics (FID, SSIM) degrade monotonically with temporal horizon; quality of generated previews is not maintained at very long look-aheads (60s).
• Evaluated only on one base VLM (Qwen2.5-VL-7B-Instruct); generalizability to other VLM architectures (e.g., LLaVA, InternVL) is claimed but not demonstrated.
• MSTP-Surgery is restricted to prostatectomy (GraSP); cross-procedure generalization is unvalidated.
• The STC requires supervised fine-tuning with domain-specific augmentation, adding overhead for new domains.

Relevance to Vision-Language Models

IG-MC directly addresses a core VLM limitation — performance degradation under long-horizon, multi-step temporal inputs — by substituting growing frame buffers with a compact state-image pair loop, making it relevant to any streaming VLM application. The multi-agent architecture (STC + hierarchical prediction agents) is a concrete instantiation of agentic VLM collaboration for structured prediction, extending LLM-agent paradigms to grounded visual prediction tasks. The benchmark provides a rigorous multi-scale evaluation surface that is currently missing for VLMs doing procedural understanding, complementing existing video-QA benchmarks (e.g., Video-MME). For researchers tracking VLMs in embodied AI and surgical robotics, IG-MC offers a modular upgrade path compatible with off-the-shelf VLMs.