Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs

🕒 Published (v1): 2026-06-28 11:42 UTC · Source: Arxiv · link

Why this paper was selected

Token merging for low-latency robotic VLM inference; practical efficiency contribution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ST-Merge is a plug-and-play, training-free framework that merges redundant visual tokens directly during the vision encoding phase using explicit 3D spatio-temporal coordinates and multi-queue parallel matching. It achieves 2× inference speedup on Qwen2.5-VL (Video QA) with only \(\sim\)1% accuracy loss, and an 8.3× speedup on the \(\pi_0.5\) VLA policy at 1024×1024 resolution with zero performance degradation on LIBERO.

Problem

High-resolution and multi-frame video inputs generate massive numbers of visual tokens, leading to \(O(n^2)\) attention complexity and prohibitive inference latency for real-time robotic control (e.g., \(\pi_0.5\) at 1024×1024 takes 2.6s per inference). Existing token-reduction methods either operate after the vision encoder (leaving encoder cost intact), use only semantic similarity without spatial structure, or ignore temporal redundancy across adjacent video frames. This forces state-of-the-art VLAs to aggressively downsample inputs (e.g., 244×244 for \(\pi_0.5\)), discarding fine-grained visual details essential for dexterous manipulation.

Method

ST-Merge operates in four stages inserted into selected shallow layers of the vision encoder (before the multimodal adapter):

1. Position-Aware Distance Computation: Each visual token is assigned a 3D spatio-temporal coordinate \(p_i = (t_i, y_i, x_i)\) from the vision encoder's feature grid. Relative position weights are computed via a Gaussian kernel \(w_{ij} = \exp(-\|p_i - p_j\|^2 / 2\sigma^2)\), constraining matching to adjacent frames/spatial regions.

2. Multi-Queue Parallel Matching: Tokens are partitioned into \(K\) sub-queues — one anchor queue \(G_1\) and \(K-1\) candidate queues. This replaces standard bipartite matching (ToMe) with one-to-many parallel matching in \(O(n)\), avoiding the serial bottleneck.

3. Weighted Spatio-Temporal Matching: The similarity between anchor token \(a_i\) and candidate \(c_j\) is \(S'_{ij} = (a_i^\top c_j) \cdot w_{ij}\), combining semantic similarity with the spatial neighborhood prior. Top-\(r\) high-confidence pairs are selected for merging. Merging uses size-weighted feature aggregation: \(v_{b_j^*} \leftarrow (s_{b_j^*} v_{b_j^*} + s_{a_i} v_{a_i}) / (s_{b_j^*} + s_{a_i})\).

4. Post-Merge Positional Correction: Recomputes RoPE embeddings for merged tokens as the weighted centroid of source tokens. The sine/cosine embeddings of the merged token are recomputed as \(\cos_{\text{new},i} = \sum_j \hat{S}_{ij} \cdot \cos_{\text{ori},j}\) (and analogously for \(\sin\)), then renormalized to preserve rotational invariance.

In VLAs (e.g., \(\pi_0.5\)), positional correction is disabled since SigLIP does not use RoPE, and only spatial (not temporal) neighborhood constraints are applied for single-frame multi-view inputs.

Key Contributions

• First plug-and-play, training-free token merging framework that operates during the vision encoding phase rather than after it, addressing the real bottleneck in VLM/VLA latency.
• Explicit 3D spatio-temporal coordinates with multi-queue parallel matching achieving \(O(n)\) complexity, enabling geometrically consistent cross-frame token fusion.
• Novel Post-Merge Positional Correction that dynamically re-estimates RoPE embeddings from the weighted centroid of merged tokens, preserving spatial awareness.
• Demonstrates 2× speedup on Qwen2.5-VL (Video QA) with marginal accuracy loss, and 8.3× speedup on \(\pi_0.5\) at 1024×1024 with zero performance degradation in both simulation and real-world deployment.

Results

• Video QA (MVP dataset, Qwen2.5-VL-7B): 28.11 TFLOPs (51% of baseline), 414.9ms inference (vs 1043.6ms baseline), 57.68% avg accuracy vs 58.73% baseline — 2× speedup with ~1% loss. Outperforms ToMe (53.41%, 848.5ms), FastV (55.68%, 719.1ms), and TempMe (55.35%, 605.8ms).
• LIBERO closed-loop VLA (\(\pi_0.5\)) at 1024×1024: 312.11ms (vs 2614.56ms baseline, 8.3× speedup), same 97.50% average accuracy.
• LIBERO at 512×512: 194.32ms (vs 607.14ms baseline), 94.16% accuracy (vs 96.67%).
• LIBERO at 244×244: 141.55ms (vs 241.21ms baseline), 94.16% accuracy (vs 95.83%).
• Ablation: Cumulative addition of Queue Partitioning (speed gain), Spatio Information (+3.12% acc), and Positional Correction (+0.78% acc) each contribute meaningfully.
• Real-world SO-ARM101 grasping tasks at 512×512 demonstrate successful deployment with low latency.

Limitations

• At lower resolutions (244×244, 512×512), ST-Merge introduces a small accuracy drop (\(\sim\)1.7–2.5%) compared to baseline, indicating the merging may discard useful information when tokens are already scarce.
• Requires deployment in global-attention layers of the vision encoder — not trivially applicable under windowed attention without modification.
• VLA variant does not benefit from the Post-Merge Positional Correction (SigLIP has no RoPE), and operates on single-frame multi-view input without exploiting temporal redundancy.
• Evaluated on only one VLM (Qwen2.5-VL) and one VLA (\(\pi_0.5\)); generalization to other architectures (e.g., LLaVA, OpenVLA) is not shown.

Relevance to Vision-Language Models

This paper directly tackles the core deployment bottleneck of VLMs on physical robots: prohibitive inference latency from dense visual token inputs. By operating during the vision encoding phase with explicit spatio-temporal awareness, ST-Merge fills a gap left by prior token-reduction methods (ToMe, FastV) that only prune after encoding or lack spatial/temporal structure. For researchers tracking VLMs, this work is significant because it demonstrates that aggressive token merging (70% reduction) can be achieved with minimal accuracy loss, and that the bottleneck for real-time VLM/VLA deployment is addressable without architectural changes or fine-tuning.