Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

🕒 Published (v1): 2026-07-06 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Reflex vs. reasoning trade-off in agentic video understanding with long-term memory — multimodal agent design signal

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Light-Omni is a multimodal agent framework that replaces iterative "detective-style" reasoning in long-video understanding with single-pass reflexive responses. It introduces dual contextual states—a non-parametric global state and a parametric latent state—that together provide global narrative context and semantically aligned retrieval without explicit reasoning chains. Compared to M3-Agent, it achieves a 2.4% accuracy gain, 12.1× speedup, and 2.6× GPU memory reduction.

Problem

Current memory-augmented video agents suffer from a semantic gap between user queries and stored memory representations (captions, summaries, sparse metadata), forcing them to compensate via iterative LLM-driven query rewriting, condition generation, and multi-turn tool invocation ("detective-style" reasoning). This incurs prohibitive latency and computational cost. The root cause is the absence of a persistent global context—agents retrieve myopically over fragmented local memories.

Method

Light-Omni operates over streaming omni-modal inputs \(I^t = \{V^t, A^t, T^t\}\) via:

Multimodal Long-Term Memory (\(M\)): Three components—user profile \(M_p\) (visual avatars, preferences), semantic memory \(M_s\) (key-value textual facts), and episodic memory \(M_e\) (timestamped omni-modal scripts with visual, auditory, and assistant turn logs).

Dual Contextual States \(S^t = \{S^t_g, S^t_l\}\):

1. Global State \(S^t_g\): A non-parametric, bounded multimodal script built from \(M_e\) via resolution-decaying hierarchical merging. When node count \(N_i \geq k+1\) (empirically \(k=8\)), the oldest \(k\) nodes are merged into a single higher-level summary, preserving recent detail and compressing distant history: $\(S^t_g = \text{Update}(M^{t-1}_e \cup \Delta M^t_e,\, k)\)$

2. Latent State \(S^t_l\): Learnable soft prompts \(P_{soft} \in \mathbb{R}^{N \times D}\) appended to \([S^t_g, I^t]\) are processed in one forward pass. Specific hidden states from soft-prompt tokens are decoded via task-specific heads into retrieval embeddings \(z^t_{ret} = \text{Pro}(h^t_{ret}) \in \mathbb{R}^{D'}\) (MLP-ReLU-MLP projection) and action trigger probabilities \(p^t_{act} = \text{Sigmoid}(\text{FC}(h^t_{act})) \in \{0,1\}^2\) for search/speech.

The retrieval embedding \(z^t_{ret}\) directly rectifies the query representation toward the memory distribution, eliminating explicit textual intermediaries.

Training: Multi-LoRA adapters for memorization/generation (NTP loss \(\mathcal{L}_1\)) and reaction (hybrid cross-entropy + InfoNCE contrastive loss \(\mathcal{L}_2\) with temperature \(\tau\), weighted by \(\lambda=2\)). 43k training samples synthesized from public long videos with injected textual/auditory instructions.

Efficiency: Feature caching (avoid re-encoding inputs used in multiple stages, −8% latency) + dynamic token pruning via feature-similarity-based redundancy pruning (−34% latency), cumulatively −42%.

Key Contributions

• Light-Omni framework: reflexive, single-pass long-video agent with structured multimodal long-term memory (profile + semantic + episodic), pluggable as a universal memory module into existing MLLMs.
• Dual contextual state design: global state for bounded historical context via resolution-decaying hierarchical merging; latent state for jointly learning retrieval embeddings and action triggers within one forward pass, eliminating iterative reasoning.
• Sleep-time consolidation: asynchronous memory CRUD updates that do not block real-time inference.
• Multi-LoRA decoupling: separate adapters for memorization, generation, and reaction to avoid gradient interference.

Results

• vs. Qwen2.5-Omni-7B (base backbone): +9.5% accuracy, 20.5× speedup, 3.3× memory reduction on VideoMME-long + LVBench average.
• vs. M3-Agent (strong reasoning agent): +2.4% accuracy, 12.1× speedup, 2.6× memory reduction.
• Absolute accuracy (VideoMME-long + LVBench avg = 58.0%): surpasses GPT-4o (48.1%), Gemini-2.0-Flash (55.8%), Qwen2.5-VL-72B (56.1%), and WorldMM-8B (≈56.9% on three-benchmark average).
• Interactive latency: 2.22s total vs. M3-Agent's 25.91s (Table 2); Light-Omni requires 1.0 LLM call per query vs. M3-Agent's 2.5 average.
• As plug-in memory: improves Qwen2.5-VL-7B by +4.9%, Qwen3-VL-8B by +2.5%, Gemini-2.0-Flash by +3.8% on combined benchmarks; provides up to 7.2× speedup and 2.5× memory reduction.
• Efficiency ablation: feature caching + redundancy pruning reduce latency by 42% with slight accuracy improvement (65.6% → 66.1% on VideoMME-long).

Limitations

• Action space restricted to search and speech; tool-use extensibility claimed with "negligible cost" but not empirically validated.
• 43k training samples are synthetically constructed; real-world distribution shift not thoroughly analyzed.
• Sleep-time consolidation requires asynchronous infrastructure; latency benefits assume offline pre-processing for benchmark evaluation.
• Hierarchical merging discards fine-grained detail from distant past by design; retrieval from very long-ago events may degrade.
• Full robustness evaluation under noisy queries is truncated in the provided text; extent of noise tolerance is partially reported.
• Backbone fixed to 7B scale (Qwen2.5-Omni-7B) in the main ablations; scalability to larger backbones is only spot-checked.

Relevance to Agentic AI / LLM Agents

Light-Omni directly challenges the dominant "reason-then-retrieve" paradigm in LLM agents by demonstrating that reflexive single-pass inference with a well-designed memory architecture can outperform multi-turn tool-calling agents while being an order of magnitude faster—a key result for anyone building latency-sensitive agentic pipelines. The dual-state design (persistent global context + parametric latent retrieval) is a transferable architectural pattern for agents operating over continuous, long-horizon streams, extending beyond video to any setting where the query-memory distribution gap degrades retrieval. The sleep-time consolidation mechanism offers a concrete template for asynchronous memory management in agents that must remain responsive while continuously learning from experience. For researchers tracking agentic efficiency and memory-augmented LLM systems, the 12×+ speedup without accuracy loss against strong reasoning baselines is an empirically significant proof point.