Skip to content

Confidence-Aware Tool Orchestration for Robust Video Understanding

🕒 Published (v1): 2026-06-25 11:37 UTC · Source: Arxiv · link

Why this paper was selected

Confidence-aware tool orchestration; solves blind trust problem in video agents under perturbations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Robust-TO is an agentic video understanding framework that addresses the "Blind Trust Problem" — the implicit assumption in video LLMs that all frames are equally reliable. It integrates per-frame quality profiling, confidence-guided tool routing, and three-tier evidence synthesis, trained end-to-end with a confidence-cost GRPO reward. On corrupted video benchmarks, it achieves 54.3% average accuracy, outperforming Gemini-2.5-Pro by 16.2%p while processing 35% fewer frames.

Problem

Existing video LLMs apply uniform frame sampling and treat all perception tool outputs as equally trustworthy, regardless of visual corruption (motion blur, glare, occlusion, low-light). Under realistic corruptions, frontier models suffer 15–30%p accuracy drops while their self-reported confidence remains unchanged — the Blind Trust Problem. Tool-augmented video agents (e.g., VideoAgent) similarly report only results, not reliability, so the LLM controller cannot distinguish informative frames from noise-corrupted ones.

Method

Robust-TO is a three-stage agentic pipeline built on a host VLM (Qwen2.5/3-VL-7B) trained with GRPO:

  1. Frame Selection via Quality Profiling. A parameter-free assess_quality tool computes per-frame disturbance scores: $\(d(f_i) = \text{mean}(d_{\text{blur}}(f_i),\, d_{\text{bright}}(f_i),\, d_{\text{occl}}(f_i))\)$ using inverse Laplacian variance, illumination deviation, and Sobel-magnitude statistics. Frames are then ranked by a reliability-relevance score: $\(s(f_i) = (1 - d(f_i)) \cdot \text{sim}(\phi(f_i), \psi(q))\)$ and the top-\(K \in [4,12]\) frames are selected adaptively.

  2. Confidence-Guided Tool Routing. The VLM decomposes the query into atomic sub-queries \(\{sq_1,\ldots,sq_m\}\) and routes each to the most corruption-robust tool (e.g., caption_frame under blur, detect_objects under occlusion). Each tool returns \((r_j, c_j)\) where: $\(c_j = c_j^{\text{intrinsic}} \times \rho(F_j), \quad \rho(F_j) = \frac{1}{\lceil n/3 \rceil}\sum_{f \in F_{j,\text{lowest}}} (1 - d(f))\)$ The input reliability \(\rho\) penalizes tool inputs containing the worst third of frames.

  3. Tiered Evidence Synthesis. Evidence is grouped into high/medium/low reliability tiers. High-tier evidence drives conclusions; medium-tier is retained only when consistent with high; low-tier is a fallback with explicit uncertainty signaling.

GRPO Training combines four rewards: $\(R_{\text{total}} = R_{\text{acc}} + w(R_{\text{subq}} + R_{\text{cc}}^{\text{total}} + R_{\text{fmt}})\)$ where \(R_{\text{cc}}(c_j, T_j) = c_j - \lambda \cdot \text{cost}(T_j)\) penalizes expensive tools on degraded inputs, and \(R_{\text{subq}}\) penalizes both over- and under-decomposition using a frozen VLM estimator \(\pi_{\text{est}}\) for target sub-query count \(m^*\).

Key Contributions

  • Formal definition of the Blind Trust Problem and a framework that addresses it without supervision over corruption type or severity.
  • A unified evidence interface returning (result, confidence, source_frames, disturbance_scores) tuples, making input reliability a first-class reasoning signal.
  • A parameter-free quality profiling tool using signal-level statistics (Laplacian variance, illumination, Sobel edges) — no training data or reference frames required.
  • A confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and tool efficiency, enabling the VLM to learn when cheap tools suffice vs. when expensive ones are warranted.
  • Demonstrated 35% reduction in processed frames (32→20.7) with simultaneous accuracy improvements.

Results

  • Clean benchmarks (UrbanVideo-Bench + VSI-Bench, 8 tasks): Robust-TO (Qwen3-VL-7B) achieves 56.4% average accuracy, vs. Gemini-2.5-Pro at 46.2% (+10.2%p) and best open-source SFT baseline (Qwen2.5-VL-7B) at 45.8% (+10.6%p).
  • Corrupted UrbanVideo-Bench (5 corruption types: MB, GN, GL, Occ, LL): Robust-TO (Qwen3-VL-7B) achieves 54.3% average accuracy, vs. Video-R1 (Qwen3-VL-7B) at 48.5% (+5.8%p) and Gemini-2.5-Pro at 38.1% (+16.2%p).
  • Smallest clean-to-corrupted accuracy gap among all compared methods.
  • Frame efficiency: Adaptive key-frame selector reduces average frames from 32 to 20.7 (−35%) and cuts per-sample inference time by >35% while improving clean accuracy by 1.6%p.
  • Paradigm ablation (Qwen3-VL-7B, UV-Bench): Direct R1 baseline: 39.5%; +sub-query decomposition: 42.8%; +tools: 49.4%; +confidence: 52.6%; +GRPO: 57.3%.
  • Reward ablation (Qwen2.5-VL-7B, UV-Bench): Full: 50.7%; w/o \(R_{\text{cc}}\): 48.4%; w/o \(R_{\text{subq}}\): 48.6%; w/o frozen estimator: 49.5%.

Limitations

  • Evaluated on only two benchmarks (UrbanVideo-Bench, VSI-Bench); broader generalization to other video domains (e.g., long-form video, medical video) is unverified.
  • Quality profiling focuses on blur, brightness, and occlusion — other real-world corruptions (compression artifacts, rain/fog, camera shake) are not explicitly modeled.
  • The frozen estimator \(\pi_{\text{est}}\) for sub-query count introduces an additional inference call and may misestimate \(m^*\) for highly complex or ambiguous questions.
  • Training required 4×A100 GPUs with rollout group size 16 for ~5k steps; computational cost of the agentic pipeline at inference is not fully characterized beyond frame count reduction.
  • Confidence calibration of individual tools (e.g., detector mean confidence, token log-probabilities) is inherited rather than learned, and may be miscalibrated for novel corruption types.

Relevance to Agentic AI / LLM Agents

Robust-TO exemplifies a key design principle for robust LLM agents: epistemic awareness of evidence quality must be propagated through the entire reasoning pipeline, not absorbed at the perception layer. The unified (result, confidence) interface is a concrete instantiation of how heterogeneous tools should communicate uncertainty to a central LLM controller — directly relevant to multi-tool agent architectures. The confidence-cost GRPO reward extends tool-use RL (building on Video-R1 / DeepSeek-R1 paradigms) to optimize not just correctness but the efficiency-reliability trade-off in tool selection, which is critical for deploying agents in resource-constrained or safety-critical settings. This work also demonstrates that RL-trained tool orchestration can outperform both SFT and much larger proprietary models, reinforcing the value of process-level reward shaping over next-token imitation for agentic tasks.