Skip to content

VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft

🕒 Published (v1): 2025-08-26 06:39 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VistaWise is a cost-effective embodied agent for Minecraft that replaces large-scale finetuning of LLMs/MLLMs with a cross-modal knowledge graph (KG) augmented by a lightweight, domain-finetuned object detection model. It achieves 33% success rate on the "obtain diamond" benchmark—the highest among non-API-based agents—while requiring only 471 annotated frames and 24 GB VRAM versus competitors needing billions of frames and terabytes of VRAM.

Problem

Existing Minecraft LLM agents either (a) rely on environment-specific APIs (MineFlayer) that limit generalization, or (b) require finetuning on massive domain-specific corpora (140B–1.6B frames, 192–23,040 GB VRAM) to embed visual and factual world knowledge. Global visual encoding introduces noise that destabilizes low-level mouse-and-keyboard (MNK) action predictions. A gap exists for agents that achieve strong performance using only visual inputs and minimal training data.

Method

VistaWise is structured around a frozen GPT-4o policy with three graph-based processes and two interaction modules:

  1. Text-modal KG construction: Online Minecraft knowledge is parsed into a static KG \(G_{\text{init}} = (V_{\text{init}}, E_{\text{init}})\) with entity nodes (e.g., "Iron Ingot") and typed relation edges (e.g., "is used to craft"). Only entity names are stored—no prose—to minimize token overhead.

  2. Cross-modal graph construction: A YOLOv10-L model finetuned on 471 frames detects two entity categories—environmental entities (bounding boxes \(\{x_e, y_e, w_e, h_e\}\)) and conditional entities (inventory icons, center coordinates \(\{x_c, y_c\}\)). Their attributes are embedded as dynamic node attributes into \(G_{\text{init}}\), yielding a vision-text KG \(G = (V, E)\) updated at each timestep \(t\).

  3. Retrieval-based pooling: Two-stage graph compression: Path Searching Pooling (PSP) retains all paths \(p \in P_{\text{P-T}}(s)\) from the player node to the task-target node; Entity Matching Pooling (EMP) further filters to nodes appearing in the task prompt \(I(s)\) or having visual attributes at time \(t\), yielding \(G'(s,t) = (V_{\text{local}}, E_{\text{local}})\), which is textualized and injected into the LLM prompt.

The agent policy is: $\(A(t) = \pi_\theta\bigl(I(s),\; G'(s,t),\; H(t),\; L\bigr)\)$ where \(H(t)\) is a LIFO memory stack of recent decisions and \(L\) is a PyAutoGUI-based desktop-level skill library enabling direct MNK control of the Minecraft client without any simulator API.

Key Contributions

  • Cross-modal KG that fuses static textual factual dependencies with dynamic per-timestep visual entity attributes, without finetuning the LLM.
  • Retrieval-based pooling (PSP + EMP) tailored to lightweight KGs where standard semantic similarity retrieval fails due to sparse text.
  • Domain-adapted object detection (YOLOv10-L on 471 frames) as the sole trainable component; reduces domain-specific training data from ≥160M frames to ~500.
  • Desktop-level PyAutoGUI skill library enabling API-free MNK control, transferable beyond Minecraft-specific simulators.
  • SOTA 33% diamond-obtainment success rate among non-API-based agents, surpassing prior best of 25%.

Results

  • Obtain diamond (9 sequential sub-goals): VistaWise 33% vs. ROCKET-1 25%, JARVIS-1 9%, VPT 15%, DEPS 1%.
  • Earlier sub-goals (crafting table through stone pickaxe): VistaWise achieves 100% on all sub-goals up to stone pickaxe; deep sub-goals (iron pickaxe onward): 73%.
  • Training efficiency: 471 frames / 24 GB VRAM vs. ROCKET-1 (1.6B frames), GROOT (1.6B frames / 384 GB VRAM), Omni JARVIS (990M tokens / 640 GB VRAM).
  • Ablation—entity attributes: Retaining only entity names vs. full node text causes no crafting failures; minor SR variation attributed to random resource distribution.
  • Ablation—retrieval: Pooling-based strategy achieves lower False Positive Rate (FPR) and False Negative Rate (FNR) than BERT similarity-based retrieval on the lightweight KG.

Limitations

  • Evaluated only in Minecraft (Vanilla 1.11.2); generalization to other open-world environments is asserted but not empirically demonstrated.
  • GPT-4o is used as the policy LLM; cost per episode is not reported, and the approach inherits GPT-4o API latency.
  • Coarse-grained distance estimation via bounding-box thresholds \(k_w, k_h\) is empirically tuned and may not transfer across resolutions or game versions.
  • 15-trial evaluation per task introduces high variance; confidence intervals are not reported.
  • The skill library is manually authored from human gameplay; its coverage limits what tasks are executable.

Relevance to Harnesses / Meta-Harnesses

VistaWise is an instantiation of a modular agent harness: it decomposes the agent loop into interchangeable subsystems (perception model, KG retrieval, memory stack, skill library) orchestrated by a central LLM policy, closely mirroring the structural pattern of meta-harnesses that coordinate heterogeneous tools and knowledge sources. The retrieval-based pooling step—dynamically constructing a context-specific sub-graph at each timestep—is analogous to a meta-harness selecting and injecting only task-relevant context from a larger knowledge store, a reusable pattern for cost-controlled grounding. The separation of "what to know" (KG + retrieval) from "how to act" (skill library) and "what happened" (memory stack) maps directly onto the layered architecture typical in harness design. For researchers building harnesses, the PSP+EMP pooling strategy and the lightweight KG construction pipeline represent concrete, low-cost techniques for injecting domain knowledge without LLM finetuning.