Skip to content

ChronoStitch: Training-Free Composition of Visual KV Memories for Long-Horizon Temporal Reasoning

đŸ•’ Published (v1): 2026-07-21 19:46 UTC · Source: Arxiv · link

Why this paper was selected

Training-free KV cache composition for long-video QA; avoids video reprocessing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

đŸ’¬ Ask ChatGPT✦ Ask Claude

TL;DR

ChronoStitch is a training-free method for composing independently stored visual KV-cache chunks from video VLMs to enable long-horizon temporal reasoning. It identifies two distinct failure modes in naive cache concatenation—positional phase collisions and missing cross-chunk attention content—and addresses each with a targeted repair. The result is 3.3× faster query-time inference than full joint re-prefilling while recovering a substantial fraction of temporal-ordering accuracy.

Problem

When a VLM processes long video in chunks and stores each chunk's KV cache independently, naive concatenation fails because: (1) each chunk's keys are rotary-encoded starting from local position zero, causing overlapping temporal phases when chunks are concatenated; and (2) later chunks were never able to attend to earlier chunks during prefill, leaving a cross-chunk content gap that persists even after positional correction.

Method

ChronoStitch applies a two-stage, training-free composition procedure at query time:

Stage 1 — Three-axis mRoPE re-basing. Each chunk's stored post-rotary keys are delta-rotated from local chunk coordinates to a global video coordinate system. Qwen2.5-VL uses a three-axis multimodal RoPE with coordinates \(p = (p_t, p_h, p_w)\) for time, height, and width. The rotation angle for frequency \(i\) is \(\alpha_i(p) = p_{\sigma(i)}\theta_i\), and because planar rotations compose additively, a stored key can be shifted from local position \(p\) to global target \(p'\) via: $\(\Delta_i = p'_{\sigma(i)} - p_{\sigma(i)}\theta_i, \quad R(\Delta)\,f(W_k x,\, p) = f(W_k x,\, p').\)$ A scalar (1D) re-indexing is explicitly shown to be geometrically wrong for visual tokens: flattening spatial patches into a scalar sequence assigns different temporal indices to patches from the same frame, encoding spatial layout as false temporal displacement.

Stage 2 — Selective cross-chunk attention repair. After re-basing, later-chunk tokens still lack the hidden-state content they would have accumulated by attending to earlier chunks. The residual content gap at layer \(\ell\) is \(\Gamma_\ell = \mathbb{E}_t \| h^{\ell,\text{joint}}_t - h^{\ell,\text{indep}}_t \|^2\). ChronoStitch identifies high-impact tokens via a key-deviation proxy: a shallow recomputation is run, and tokens whose layer-1 keys shift most are scored by \(\delta_t = \|k_t^{1,\text{recomp}} - k_t^{1,\text{stored}}\|^2\). The top \(\rho\) fraction of later-chunk visual tokens are then fully recomputed with causal attention over the entire composed cache, and their new KV pairs are scattered back. At \(\rho = 0.35\), this recomputes ~25% of the composed visual cache, and the vision tower is never rerun.

Key Contributions

  • Formal identification of two independent failure modes in visual KV-cache composition: positional phase collision and cross-chunk content gap.
  • Training-free three-axis delta rotation of stored post-rotary keys to a global mRoPE coordinate frame, achieving machine-precision layer-0 key reconstruction (relative MSE \(3.6 \times 10^{-8}\)).
  • Proof-of-concept that scalar 1D re-indexing is geometrically defective for video tokens and empirically worse than naive concatenation at layer 0.
  • Oracle-free key-deviation proxy for ranking which later-chunk tokens most need cross-chunk attention repair.
  • Controlled order-swap probe demonstrating that positional correction alone recovers 0/6 swapped pairs; adding \(\rho = 0.35\) repair recovers 6/6.

Results

  • TempCompass temporal split (N=590 MC questions, Qwen2.5-VL-3B):
  • ChronoStitch: 54.1% overall, 54.0% event ordering, 54.2% attribute change
  • Three-axis re-basing only: 49.8% / 46.7% / 53.1%
  • Scalar1D re-basing: 49.5% / 47.4% / 51.7%
  • Naive concatenation: 49.3% / 47.0% / 51.7%
  • Joint prefill ceiling: 63.9% / 60.9% / 67.0%
  • ChronoStitch improves +4.8 pp overall and +7.0 pp on event ordering vs. naive concatenation; recovers ~half the naive-to-joint gap on event ordering.
  • Controlled order-swap probe: ChronoStitch (\(\rho=0.35\)) achieves 100% accuracy (6/6 flipped pairs), matching joint prefill; re-basing only achieves 41.7%.
  • Query-time efficiency (12 videos): ChronoStitch mean 748 ms vs. joint re-prefill 2411 ms — 3.3Ă— speedup (range 3.19–3.43Ă—), recomputing only ~25% of the composed visual cache.
  • Layer-0 key fidelity: Three-axis re-basing: cosine similarity 1.0000, rel. MSE \(3.6\times10^{-8}\); naive: cosine 0.9945, rel. MSE \(9.7\times10^{-3}\); scalar1D: cosine 0.9921, rel. MSE \(1.3\times10^{-2}\).

Limitations

  • Reader model is small (3B parameters); joint prefill ceiling is only 63.9%, compressing observable margins and making it difficult to isolate the effect of scalar vs. three-axis re-basing on downstream QA.
  • Optimal repair fraction \(\rho\) was selected from a small controlled probe; it may not generalize across model scales, chunk lengths, or video durations.
  • Efficiency is reported as wall-clock latency on Apple M-series hardware only; no hardware-independent FLOP or memory analysis is provided.
  • The downstream QA gap between scalar and three-axis re-basing is not yet large enough to be unambiguously measurable on the 3B model without the repair stage.
  • Experiments use only Qwen2.5-VL-3B; generalization to other VLMs or larger mRoPE architectures is untested.

Relevance to Vision-Language Models

ChronoStitch directly targets a practical bottleneck in deploying VLMs on long video: efficient KV-cache reuse across independently prefilled chunks, which matters for any system that pre-indexes video before knowing the query. The paper's decomposition of positional vs. content errors in KV composition is a concrete diagnostic framework applicable to any VLM using multimodal RoPE (e.g., Qwen2-VL, Qwen2.5-VL), and the finding that positional correction is necessary but insufficient will inform future cache management designs. The scalar-vs-three-axis re-basing analysis is particularly relevant to researchers extending RoPE-based VLMs to video, as it shows that collapsing spatial patch order into a temporal dimension introduces systematic, measurable errors. The method connects directly to emerging work on streaming video KV management (ReKV, StreamKV, HERMES, DSCache) by providing a principled composition step that those systems currently omit.