Skip to content

Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NeuRO is a hybrid learning-to-optimize framework for multi-object visual navigation that couples a neural perception module with a downstream robust optimization planner. It addresses data scarcity and partial observability by converting noisy visual predictions into convex uncertainty sets via PICNN-based conformal calibration, then solving a tractable robust LP. It achieves state-of-the-art on MultiON benchmarks, particularly in generalization to unseen environments.

Problem

Neural network-only navigation agents overfit in low-data regimes (e.g., search-and-rescue) because increasing architectural complexity exacerbates the problem. Naively coupling networks with optimization fails for two reasons: (1) noisy network predictions in data-scarce settings destabilize the optimizer, and (2) partial observability in embodied environments means the global information required to formulate the optimization problem is unavailable.

Method

NeuRO integrates four tightly coupled components:

  1. Neural perception module: A CNN+GRU processes egocentric RGB-D images and an egocentric map, producing per-object state features ft and a preliminary policy π(anet_t | ft).

  2. PICNN-based conformal calibration (learning→planning interface): Raw network features are passed through a Partially Input Convex Neural Network (PICNN) to define a nonconformity score g(x, M) that is convex in the transition matrix M. An uncertainty set Ω(x) = {M | g(x,M) ≤ q} is formed, where threshold q is the ⌈(N+1)(1−α)⌉-th quantile of calibration scores, guaranteeing 1−α marginal coverage. This yields a tractable, convex uncertainty set directly parameterizing the downstream optimizer.

  3. Robust optimization planner: MultiON is cast as a pursuit-evasion game on a discrete grid H. The agent maintains prior/posterior belief states β, α over object locations; the objective maximizes discounted capture belief worst-case over M ∈ Ω(x). The min-max problem is dualized via KKT/strong duality into a linear program (U-MON*, S-MON*) solvable at inference.

  4. Planning→learning feedback: The optimizer's output feeds back in two ways: (a) an action blending a_t = λ·anet_t + (1−λ)·atask_t where atask_t is derived from the optimal trajectory p*; (b) an optimization-derived task reward rttask combined with environment reward via Goal Vector Method (GVM) for Pareto-optimal multi-reward training. Gradients flow through the LP via implicit differentiation of KKT conditions.

Key Contributions

  • First end-to-end learning framework coupling deep neural networks with downstream robust optimization for embodied visual navigation tasks.
  • PICNN-based conformal calibration method that converts unreliable visual predictions into convex uncertainty sets with provable marginal coverage guarantees.
  • POMDP-as-pursuit-evasion-game formulation that handles partial observability and is reformulated into a tractable LP via strong duality.
  • Dual planning↔learning interface: optimizer outputs refine both actions and reward signals; gradients propagate back through KKT conditions.
  • Demonstrated task adaptability: switching between U-MON and S-MON requires only changing the downstream LP, not retraining the network.

Results

  • S-MON m=2: NeuRO achieves 80% Success / 86% Progress / 66% SPL / 72% PPL vs. HTP-GCN (76/84/60/70) and OracleEgoMap (64/71/49/54).
  • S-MON m=3: NeuRO achieves 62% Success / 72% Progress / 40% SPL / 47% PPL vs. HTP-GCN (57/70/36/45) and Lyon (57/70/27/33).
  • U-MON m=2: NeuRO 62% Success / 66% Progress / 52% SPL / 56% PPL vs. OracleEgoMap 54/62/38/44.
  • U-MON m=3: NeuRO 45% Success / 53% Progress / 45% SPL / 50% PPL vs. OracleEgoMap 36/42/33/38.
  • Cross-task generalization (Table 5): NeuRO's average performance drop when evaluated out-of-training-distribution is substantially smaller than OracleEgoMap and Lyon baselines.
  • Ablation – coverage level: Tighter uncertainty sets (α=0.01) give 80% Success / 66% SPL; looser (α=0.20) degrades to 66% Success / 52% SPL.
  • Ablation – action blend λ: Best at λ=0.2 (80% Success); excessive λ=0.4 drops to 52%.

Limitations

  • PICNNs approximate uncertainty as convex sets; inherently non-convex uncertainties in complex visual scenes cannot be captured, making convexity a fundamentally approximate assumption.
  • The optimization grid H and horizon Ï„ are discrete abstractions that coarsen real 3D navigation geometry; increasing them raises inference time (E=5, Ï„=6 costs ~1.08s per step).
  • The approach has only been evaluated on MultiON benchmarks in simulation; sim-to-real transfer is not addressed.
  • Conformal calibration assumes i.i.d. samples from the implicit training distribution; distribution shift at test time may violate coverage guarantees.

Relevance to Agentic AI / LLM Agents

NeuRO exemplifies a broader paradigm shift from purely neural agents toward hybrid neuro-symbolic agents that embed structured, uncertainty-aware optimization directly into the decision loop—relevant to any agentic system that must plan under partial observability with limited data. The conformal calibration + robust LP pattern (learn uncertainty → optimize robustly → backprop through KKT) is a principled template for grounding uncertain perceptual predictions in formal task constraints, addressing a core failure mode of LLM-based agents that rely on black-box neural outputs for sequential planning. The result that task-specific optimization structure transfers across goal-count variations without retraining mirrors the composability goals of modular agent architectures. For long-horizon embodied agents specifically, this work demonstrates that explicit constraint representation outperforms additional network complexity under data scarcity.