Event-VLA: Action-Conditioned Event Fusion for Robust Vision-Language-Action Model¶

🕒 Published (v1): 2026-06-28 13:19 UTC · Source: Arxiv · link

Why this paper was selected

Event camera fusion addresses RGB degradation in VLA; novel sensor modality for embodied AI

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Event-VLA augments pretrained Vision-Language-Action (VLA) models with event camera streams through an action-conditioned gated cross-attention interface that keeps event tokens outside the semantic backbone, preventing disruption of RGB-language priors while providing illumination-robust motion cues. On the severely degraded (near-dark) LIBERO-Cross benchmark, it achieves 95.6% success rate versus 61–70% for RGB-only baselines, with only 2.2ms latency overhead.

Problem¶

Existing VLA models assume well-lit, stable indoor settings with high-quality RGB observations, but real-world robotic manipulation faces illumination shifts, motion blur, and sensor noise that degrade semantic grounding and action stability. Event cameras naturally provide high-dynamic-range, motion-sensitive signals under such conditions, but the interface question of how to asynchronously inject high-frequency event streams into pretrained VLAs without corrupting their global semantic representations is unsolved.

Method¶

PREI (Physical Residual Event Integration): Compresses sparse, asynchronous event streams ($e_i = (x_i, y_i, \tau_i, p_i)$) into a compact three-channel residual map $\mathbf{E}_t^{\text{prei}} = [\mathbf{E}_t^{\text{ins}}, \mathbf{E}_t^{\text{sal}}, \mathbf{E}_t^{\text{per}}] \in [0,1]^{H \times W \times 3}$ capturing instantaneous, salient, and persistent motion traces via exponential-decay activity accumulation and local normalization.
Action-conditioned event interface: Event tokens $\mathbf{Y}_t^e$ are never fed into the pretrained VLA backbone $\mathcal{F}_{\text{vla}}$. Instead, they are fused after backbone processing through gated cross-attention: $$\mathbf{Z}_t^{\text{fused}} = \text{Norm}\left(\mathbf{Z}_t^{\text{lm}} + \sigma(\mathbf{f}_{\text{gate}}(\cdots)) \odot \text{CrossAttn}(\mathbf{Z}_t^{\text{lm}}, \mathbf{Y}_t^e)\right)$$ where $\sigma$ is a learnable gate produced from both VLA hidden states and event-attended features.
Query-guided token routing: Separates fused tokens via three learnable query types (common, action, event). Action queries route task-relevant tokens to the action head; event queries route event-relevant tokens to an auxiliary future-PREI prediction head that regularizes training.
Training objective: $\mathcal{L} = \mathcal{L}_{\text{act}} + \lambda_{\text{evt}}\mathcal{L}_{\text{evt}} + \lambda_{\text{deriv}}\mathcal{L}_{\text{deriv}}$, combining $\ell_1$ action loss, masked $\ell_1$ future-PREI prediction loss, and first-order derivative consistency on PREI maps.

Key Contributions¶

Formulates degraded-visibility VLA manipulation as an event-to-VLA interface problem rather than a perception problem — emphasizes how events connect to the policy, not just event representation.
Proposes Event-VLA with an action-conditioned interface that structurally separates event tokens from the pretrained semantic backbone, using gated cross-attention and query-guided routing.
Introduces PREI, a lightweight decomposition of event streams into instantaneous/ salient/ persistent action-time physical residual maps, optimized for VLA compatibility via feature-level distillation.
Demonstrates that the interface design (action-conditioned routing) matters as much as the representation (PREI): query routing achieves 95.6% SR with +2.2ms latency, while unified encoding matches SR but costs +62.9ms.

Results¶

LIBERO (normal visibility): 96.5% average SR, comparable to SOTA RGB-only policies (MM-ACT 96.3%, π0 94.2%).
"Ours w/o event" ablation: 96.2% average SR — disabling the event pathway at inference does not degrade pretrained VLA performance, confirming the interface preserves RGB-language priors.
LIBERO-Cross LL-Severe (near-dark): 95.6% SR — substantially outperforming π0 (61.9%), OpenVLA-OFT (61.2%), and MM-ACT (69.6%).
LIBERO-Cross LL-Dark: 96.5% SR vs. next best MM-ACT (93.0%).
LIBERO-Cross LL-Mild: 96.1% SR (marginal gain; next best MM-ACT 95.9%).
Real-world Franka deployment (4 tasks, 10 trials each): 65.0% average SR vs. π0 (48.3%) and OpenVLA-OFT (46.7%).
Interface ablation (LL-Severe): PREI (95.6%) > time surface (91.2%) > no event (60.6%). Query routing (95.6%, +2.2ms) ≈ unified encoding (95.1%, +62.9ms) but 30× more efficient.

Limitations¶

LIBERO-Cross uses an RGB-to-event simulator (v2e), which may not capture real event-camera noise, triggering dynamics, trigger jitter, or calibration errors.
Real-world evaluation is small-scale: 4 manipulation tasks, 10 trials per lighting condition, single robot platform (Franka Research 3).
Requires additional event-camera hardware (DAVIS346) and synchronization, increasing deployment complexity and cost.
Addresses only illumination-related degradation, not other failure modes (occlusion, object variability, long-horizon compounding errors).

Relevance to Vision-Language Models¶

This paper is directly relevant because VLA models — the embodied-action branch of VLMs — inherit VLM backbones (here, Prismatic-7B / OpenVLA) and thus their sensitivity to visual degradation. The core architectural finding — that auxiliary sensory modalities should be routed through task-specific (action) pathways rather than naively fused into the global semantic token space — is a general design principle that applies to any VLM being deployed in uncontrolled environments. For researchers tracking VLMs, this work highlights a practical robustness gap and offers a light-weight, interface-level fix that preserves the value of large-scale pretraining.