Skip to content

ActiveVOO: Value of Observation Guided Active Knowledge Acquisition for Open-World Embodied Lifted Regression Planning

๐Ÿ•’ Published (v1): 2025-01-01 ยท Source: NeurIPS ยท Venue: NeurIPS 2025 ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

ActiveVOO is a zero-shot, open-world embodied planning framework that uses lifted regression to identify task-relevant objects and a decision-theoretic Value of Observation (VOO) metric to prioritize which objects a VLM should actively sense. By coupling symbolic backward-chaining plans with LLM/VLM commonsense probability estimates, the agent avoids exhaustive passive perception. It achieves 0.86 success rate on visual ALFWorld, outperforming fine-tuned VLM baselines without any task-specific training data.

Problem

Embodied agents in open-world settings face partial observability and an unbounded, task-irrelevant object space. Passive perception strategies (collect everything visible) suffer combinatorial explosion in object/relation enumeration. Prior active approaches either assume a known target object category or rely on generic information-gain metrics that skip the crucial step of determining which objects are relevant to the current goal before deciding which to sense first.

Method

ActiveVOO decomposes active knowledge acquisition into two steps:

  1. Lifted Regression for Subgoal Generation. Starting from a PDDL goal, the agent applies lifted backward search to derive a set of abstract subgoals {(g_i, ฯ€_i)}. Each g_i is a first-order formula over typed variables (e.g., isPotato(?x) โˆง isClean(?x) โˆง isFridge(?z)), deferring variable instantiation until objects are observed.

  2. Object/Relation Description Extraction. Unary predicates per variable yield object descriptions O; binary-or-higher predicates yield relational descriptions R. These form the candidate sensing targets.

  3. Sensing Belief via LLM Logits. For each description, the agent queries GPT-4o with a yes/no question in a household context and uses the softmax of logit("Yes") vs. logit("No") (averaged over k=5 samples) as a commonsense probability p_o or p_ฯ.

  4. Value of Observation (VOO). Utility U(s) is the maximum over subgoals of the product of sensing-success probabilities for all still-unknown descriptions. VOO for sensing object o is p_o * (U(s'_+) โˆ’ U(s'_-)), i.e., the expected utility gain minus the baseline of not sensing. The agent greedily selects the highest-VOO target, navigates there, queries the VLM (GPT-4o) to detect the top-5 candidate objects, updates the knowledge state, and repeats until a lifted subgoal is fully groundable.

Key Contributions

  • Lifted regression applied to open-world active sensing: subgoals identify what types of objects are task-relevant before any grounding is attempted.
  • Value of Observation (VOO): a decision-theoretic utility measure (distinct from VOI โ€” sensing causally enables the outcome here) for prioritizing among candidate sensing targets.
  • LLM logit-based commonsense belief elicitation to estimate per-object and per-relation sensing probabilities without domain-specific training data.
  • State-of-the-art zero-shot performance on visual ALFWorld (0.86 success rate), surpassing both fine-tuned VLMs (EMMA-3: 0.37, BUTLER: 0.26) and zero-shot VLMs (GPT-4o Vision: 0.08, LLaVA-13B: 0.11).

Results

  • ActiveVOO overall: 0.86 success rate, avg. episode length 15.3 steps (30-step budget, zero-shot, single trial, visual ALFWorld).
  • vs. GPT-4o Vision (zero-shot): 0.86 vs. 0.08 โ€” ~10ร— improvement.
  • vs. LLaVA-13B (zero-shot): 0.86 vs. 0.11.
  • vs. EMMA-10 (fine-tuned, 10 trials): 0.86 vs. 0.82 โ€” ActiveVOO matches with zero data and one trial.
  • vs. REFLEXION-10 (LLM-only, 10 trials): 0.86 vs. 0.91 โ€” only method that beats ActiveVOO requires 10 trials on the same environment with textual input.
  • Ablations (Table 2):
  • Exhaustive (passive) object acquisition: 0.22 (โ†“0.64), episode length 25.4.
  • Goal-directed (VLM queried with task goal only): 0.48.
  • LLM-subgoal guided (no lifted regression): 0.47.
  • Without VOO (use belief probability rank only): 0.75, length 19.2.
  • Without VOO + object partition: 0.68.
  • Replace GPT-4o VLM with LLaVA-13B: 0.82.
  • Replace GPT-4o LLM with GPT-3.5: 0.77.

Limitations

  • Evaluated only on ALFWorld simulator; real-robot transfer requires additional fine-tuning to handle perception noise and actuation uncertainty.
  • Does not model user preferences or contextual constraints in goal specification, which could alter the agent's utility function.
  • VOO computation assumes independent sensing outcomes across object/relation descriptions, which may not hold when objects co-occur or share spatial context.
  • Greedy VOO selection (one object at a time) may be suboptimal compared to planning over sensing sequences jointly.
  • Relies on GPT-4o API calls for both LLM belief elicitation and VLM object detection; computational cost and latency scale with the number of sensing rounds.

Relevance to Vision-Language Models

ActiveVOO exemplifies a hybrid paradigm where VLMs (GPT-4o) serve not as end-to-end planners but as targeted perceptual oracles called only on VOO-prioritized queries, sidestepping the notorious challenge of VLMs hallucinating or getting overwhelmed by cluttered scenes. The work directly benchmarks zero-shot VLMs (GPT-4o Vision, LLaVA-13B) and fine-tuned VLMs (InstructBLIP, LLaMA-Adapter, EMMA) on visual ALFWorld, providing clear evidence that raw VLM capability is far weaker than structured symbolic guidance + selective VLM querying. For researchers tracking VLMs, this paper demonstrates that the key bottleneck is not VLM perception quality per se but the query formulation โ€” lifted regression produces compact, semantically precise prompts that let the VLM succeed where open-ended prompts fail. It also connects to the broader literature on LLM/VLM-as-planner vs. LLM/VLM-as-module debates in embodied AI.