Skip to content

Native Active Perception as Reasoning for Omni-Modal Understanding

🕒 Published (v1): 2026-06-17 17:59 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; active-perception reasoning cuts compute for long-video agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniAgent is a native omni-modal agent that reformulates long-video understanding as a POMDP-based Observation-Thought-Action (OTA) cycle, selectively distilling audio-visual cues into persistent textual memory so that reasoning complexity scales with query difficulty rather than video duration. Trained via two-stage agentic SFT + a novel RL objective (TAURA), the 7B model achieves state-of-the-art results among open-source models across ten benchmarks, including outperforming Qwen2.5-VL-72B on LVBench (50.5% vs. 47.3%) using 73% fewer frames.

Problem

Passive "watch-it-all" multimodal models process video frames uniformly, causing computational cost to grow super-linearly with video duration and making hour-long video understanding intractable. Prior agentic workarounds either (a) delegate perception to external specialist modules—severing gradient flow—or (b) still require a global pre-scan before acting, failing to truly decouple reasoning cost from video length.

Method

OmniAgent models video exploration as a POMDP with state \((M_k, E_k)\), where \(M_k\) is a persistent textual memory and \(E_k\) is a transient multimodal percept discarded after each turn. At each step \(k\), a single policy \(\pi_\theta\) autoregressively generates an OTA triplet \((O_k, T_k, A_k)\):

\[( O_k, T_k, A_k ) \sim \pi_\theta(\cdot \mid M_{k-1}, E_{k-1})\]

The action space \(\mathcal{A} = \{a_\text{frames}, a_\text{audio}, a_\text{clip}, a_\text{answer}\}\) lets the agent retrieve frames, audio segments, or synchronized clips on demand; after each action the raw percept is purged, keeping media overhead constant.

Agentic SFT bootstraps the policy on 58K trajectories synthesized via best-of-N teacher exploration with dual-stage quality control: (1) outcome verification (exact match / IoU ≥ 0.5 / MRA ≥ 0.5), and (2) a GPT-4o rationality audit on a 5-point Likert scale (threshold ≥ 3/5) to filter lucky guesses.

TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage) addresses the Advantage Homogenization problem in multi-turn GRPO, where a single scalar advantage is broadcast uniformly across all turns. TAURA rescales the advantage per turn \(k\) by its normalized mean token entropy \(H_{i,k}\):

\[\hat{A}_{i,k} = A_i \cdot \frac{H_{i,k}}{\frac{1}{N_G}\sum_{j=1}^{G}\sum_{m=1}^{K_j} H_{j,m}}\]

This amplifies credit for high-entropy "discovery" turns and penalizes confused guessing, with empirical support that 79.2% of critical branching turns have above-average entropy.

Key Contributions

  • First end-to-end native omni-modal agent (single model handles perception + reasoning + action) formulated as a POMDP with persistent textual memory, decoupling context cost from video duration.
  • Agentic SFT pipeline: best-of-N trajectory synthesis with error-recovery traces and dual-stage quality control (outcome + rationality).
  • TAURA: entropy-steered, turn-level credit assignment for multi-turn RL, resolving advantage homogenization in GRPO.
  • Positive test-time scaling: performance improves with more reasoning turns (+6.2% on VideoMME-Long).
  • State-of-the-art across ten benchmarks (video understanding, audio-visual reasoning, temporal grounding) at 7B scale.

Results

  • LVBench (long video, up to 4101 sec): 50.5% — surpasses Qwen2.5-VL-72B (47.3%) with 73% fewer frames; +7.5% over Qwen2.5-Omni-7B baseline.
  • MLVU: 71.1% (+5.9% over baseline); best open-source.
  • VideoMME-Long: 59.6% (+4.8% over baseline).
  • VSI-Bench (spatial reasoning): 48.4% (+12.9% over baseline).
  • Minerva: 41.4% (+8.0% over baseline).
  • DailyOmni (audio-visual): 64.8% (+4.7% over baseline).
  • OmniVideoBench: 37.1% (+7.8% over baseline).
  • LongVALE (temporal grounding): 39.1 IoU (+33.4% absolute over Qwen2.5-Omni baseline of 5.7).
  • VUE-TR: 36.5 IoU (+33.0% absolute), surpassing GPT-4o and Gemini-2.5-Pro.
  • Test-time scaling confirmed: additional reasoning turns monotonically improve VideoMME-Long by +6.2%.

Limitations

  • RL training restricted to videos under 300 seconds; generalization of TAURA to very long video trajectories is not directly validated.
  • Relies on Qwen2.5-Omni-7B as backbone; audio capabilities are tied to that base model's omni-modal pretrain.
  • Maximum context window of 64K tokens caps the number of consolidated observations; extremely long or dense videos may still hit this ceiling.
  • No ablation reported on the interaction between the SFT corpus size (58K) and RL sample efficiency.
  • Evaluation is primarily multiple-choice or IoU-based; open-ended generation quality is not assessed.

Relevance to Agentic AI / LLM Agents

OmniAgent exemplifies the shift from tool-augmented LLM pipelines toward native agentic models—a single model that reasons, plans, and perceives without delegating to external modules, directly relevant to the question of how to build general-purpose embodied or video-grounded agents. The POMDP formulation with persistent memory and context purging is a concrete architecture for managing long-horizon agent state under token-budget constraints, a central challenge in agentic systems. TAURA's turn-level credit assignment addresses a fundamental training problem for any multi-turn RL agent (advantage homogenization in GRPO), making it applicable beyond video to tool-use and dialogue agents. The positive test-time scaling property—more reasoning turns → better performance—validates inference-time compute scaling as a lever for agentic reasoning, complementing trends in chain-of-thought and process-reward models.