Seek to Segment: Active Perception for Panoramic Referring Segmentation

🕒 Published (v1): 2026-07-02 17:56 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Active perception for panoramic segmentation; bridges VLMs and embodied AI agent perception

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Active Panoramic Referring Segmentation (APRS), a new task requiring an agent to actively explore 360° environments to locate and segment a language-specified object. The proposed agent, PanoSeeker, integrates a VLM with EgoSphere—an explicit spatial-visual memory maintained as an equirectangular panorama (ERP) canvas—to plan efficient, non-redundant search trajectories. Training combines supervised fine-tuning on expert-annotated trajectories with GRPO reinforcement learning optimizing exploration efficiency.

Problem

Existing referring segmentation models assume the target is visible in a static, fixed-view image, making them inapplicable to Embodied AI settings where targets may be anywhere in a continuous 360° environment. Explicit 3D reconstruction (point clouds, Gaussian Splatting) is computationally prohibitive; naive 2D methods applied to equirectangular panoramas suffer from geometric distortion and loss of spatial context; and sequential perspective-view approaches accumulate redundant exploration without global map awareness.

Method

Task formulation. At each step \(t\), the agent observes \(V_t = \text{Proj}(\mathcal{E}, \theta_t, \varphi_t, \psi)\) via gnomonic projection (\(\psi_h = 120°\), \(\psi_v = 90°\)) and predicts action \(a_t = (\Delta\theta_t, \Delta\varphi_t)\) or \([\text{STOP}]\).

EgoSphere memory. A fixed-resolution ERP canvas \(M \in \mathbb{R}^{H \times W \times 3}\) is updated via inverse gnomonic projection at each step: $\(M_t = M_{t-1} \oplus \text{Proj}^{-1}(V_t, \theta_t, \varphi_t, \psi)\)$ Unexplored regions remain zero-valued. The canvas is augmented with a crosshair, a 30°-interval lat/lon grid, and the rendered trajectory path, converting search into a visual "fill-in-the-blank" task on the 360° canvas. Context length is constant regardless of trajectory length.

PanoSeeker (Qwen3-VL-8B-Instruct + LoRA, rank \(r=64\)). The VLM jointly attends to the current local view \(V_t\), the annotated EgoSphere map \(M_t\), and language instruction \(I\) to predict the next action.

Training. (1) SFT via negative log-likelihood over expert trajectories \(\mathcal{D}_\text{expert}\). (2) GRPO (\(G=8\), \(\epsilon=0.2\), 6,000 steps) with a per-step efficiency reward \(R_\text{eff} = \gamma[\text{dist}(V_{t-1}, V^*) - \text{dist}(V_t, V^*)] - \eta\) and a terminal reward \(R_\text{term} = R_\text{base} + \alpha \cdot \text{IoU}\) on success or \(-R_\text{pen}\) on failure.

Active Alignment and Segmentation. On \([\text{STOP}]\), PanoSeeker predicts a bounding box; SAM-3 is invoked with an active angular adjustment \((\Delta\theta^*, \Delta\varphi^*)\) to center the target and produce mask \(S\).

Key Contributions

• Defines the APRS task: active, language-guided search and segmentation in continuous 360° environments.
• Constructs the APRS benchmark: 7,420 samples across 4,971 scenes with four spatial referring expression types (EGO, UNIQ, ALLO, MULTIHOP) and multi-dimensional evaluation (SR, AS, SPL, mIoU).
• Proposes EgoSphere, a constant-context spatial visual memory that accumulates observations into a geometrically consistent annotated ERP map with spatial visual prompting.
• Provides an expert-annotated SFT trajectory dataset with memory timelines.
• Applies GRPO with geodesic efficiency and IoU-conditioned terminal rewards to explicitly optimize exploration policy.

Results

The results table is partially truncated in the provided text; only static-method baselines are fully visible: - Static methods (processing full ERP in one pass): SR 46.7–66.2%, AS 1.0 (single step), mIoU 34.7–47.7%. - The paper states PanoSeeker achieves "a significant leap in exploration efficiency (SPL and AS)" over all three paradigms (static, heuristic scanning, active VLM agents), outperforming all adapted state-of-the-art baselines on the APRS benchmark. - Headline numbers for PanoSeeker itself are not visible in the provided text due to truncation.

Limitations

• Benchmark is simulation-only (gnomonic projection from static panoramas); real-world continuous capture introduces motion blur, exposure variation, and latency not modeled here.
• Maximum episode length is capped at \(T=20\) steps; failure modes at harder spatial configurations (large scenes, dense clutter) are not characterized.
• EgoSphere resolution is fixed at 1024×512; fine-grained objects at the edges of FoV projections may be blurred or misregistered by the inverse gnomonic mapping.
• Relies on SAM-3 and Qwen3-VL-8B-Instruct as frozen/fine-tuned components; performance is coupled to their failure modes (hallucination in grounding, SAM boundary errors).
• Annotation used GPT-5.2 for semantic cross-checking, introducing potential label bias from that model's perception.

Relevance to Vision-Language Models

This work extends VLMs from passive single-image grounding into embodied active-perception agents, demonstrating that a VLM backbone (Qwen3-VL-8B) fine-tuned with expert trajectories and RL can plan spatially coherent 360° search without explicit 3D reconstruction. The EgoSphere design is directly relevant to researchers studying how VLMs can be grounded in egocentric spatial memory—a gap between current VLM capabilities and real-world robotic deployment. The GRPO-based reward formulation (geodesic efficiency + IoU) provides a concrete template for reinforcement learning over embodied VLM agents where both navigation and perception quality must be jointly optimized. The APRS benchmark fills a niche between pure navigation tasks (no pixel-level output) and static referring segmentation (no active exploration), offering a new evaluation axis for spatially capable VLMs.