Dual-Scale World Memory for LLM Agents towards Hard-Exploration Problems¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Dual-scale world memory enables LLM agents in hard-exploration tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GLoW (Global-Local World Memory) introduces a dual-scale textual memory framework that enables LLM agents to tackle hard-exploration problems by maintaining a value-ranked trajectory frontier for principled global state selection and a Multi-path Advantage Reflection (MAR) mechanism for dense local progress signals. On the Jericho text-game benchmark, GLoW achieves new state-of-the-art performance among LLM-based approaches across 7 of 10 games while requiring 100–800× fewer environment interactions than RL baselines.

Problem¶

LLM agents (ReAct, Reflexion) support local trial-and-error but lack mechanisms for long-term accumulation of exploration knowledge. Hard-exploration problems—characterized by sparse rewards, combinatorial state-action spaces (e.g., $O(697^5) \approx 1.64 \times 10^{14}$ actions per step in Zork1), and deceptive local optima—expose this gap: existing LLM agents score far below RL/MCTS methods on Jericho while RL methods require hundreds of thousands of environment interactions. No prior LLM framework integrates both sustained global learning and advantage-driven local exploration.

Method¶

GLoW is instantiated on top of the Go-Explore framework (alternating select and explore phases) and adds two complementary world memory modules:

Global World Memory (state selection). A value-ranked trajectory frontier $F = \{\tau_1, \ldots, \tau_k\}$ retains the top-$k$ trajectories by $v(\tau_i) = \max_{t} \sum_{j=1}^{t} r_j^i$. An LLM $g_{\text{LLM}}$ analyzes all frontier trajectories to produce: $$W_{\text{global}} = g_{\text{LLM}}(F) = \{(s_i, v_i, v_i')\}$$ where $v_i$ is achieved value and $v_i'$ is LLM-estimated future potential at critical states (bottlenecks, milestones). This implements a semantic analog of UCB: $v_i'$ captures optimism under uncertainty by reasoning about why trajectories fail. An alignLLM operation then scores each archived state against $W_{\text{global}}$ to select the next exploration start $s_{\text{next}}$.

Local World Memory (exploration via MAR). From $s_{\text{next}}$, $n$ trajectories are sampled sequentially. After each trajectory $\tau_i$, Multi-path Advantage Reflection compares trajectories against the frontier (stable reference baseline) to infer semantic advantages at 2–4 critical states: $$W_{\text{local}} = \text{MAR}(\{\tau_1,\ldots,\tau_i\}, F) = \{(s_j^*, A_{s_j^*})\}$$ $W_{\text{local}}$ guides the next trajectory via $\pi_{\text{explore}}(a_t | s_t, h_t) = \text{AgentLLM}(s_t, h_t, W_{\text{local}}, T_s, F)$. MAR densifies sparse rewards by comparing divergent outcomes across multiple rollouts from the same state, analogous to advantage estimation $A(s,a) = Q(s,a) - V(s)$ in RL but computed through LLM reasoning rather than numerical subtraction. Hyperparameters: $n=3$ exploration trajectories, $k=5$ frontier size.

Key Contributions¶

GLoW framework: dual-scale world memory unifying global and local learning for hard-exploration in LLM agents.
Value-ranked trajectory frontier: preserves full temporal context of high-value episodes, enabling semantic UCB-style decomposition into achieved vs. potential value.
Multi-path Advantage Reflection (MAR): derives pseudo-dense advantage signals from sparse feedback via multi-trajectory LLM comparison, reducing high-variance Q-value estimation.
Sample efficiency: 100–800× fewer environment interactions than RL baselines (1,000 vs. up to 1.6M steps).
New LLM SOTA on Jericho: best LLM performance on 7/10 games; surpasses all prior methods (including RL/MCTS) on 4/6 difficult and extreme games when scaled to stronger LLMs.

Results¶

Zork1: GLoW 73.0 vs. ICRL 51.7 (best LLM prior), XTX 103.4 (best RL, 800,000 interactions).
Ludicorp: GLoW 73.7 vs. ICRL 32.0 (best LLM prior).
Enchanter: GLoW 61.7 vs. IGE 50.0 (best LLM prior), MC-DML 20 (MCTS).
Deephome: GLoW 75.0 vs. IGE 71.3 (best LLM prior), XTX 77.7 (best RL).
Zork3: GLoW 4.3 vs. IGE 3.7 (best LLM prior).
Balances: GLoW 16.7 vs. ICRL 11.7 (best LLM prior).
GLoW achieves LLM SOTA on 7/10 games; Pentari and Detective show similar performance to LLM baselines (30.0 and partial data).
All LLM methods use 1,000 interactions; RL methods use 100,000–1,600,000.

Limitations¶

Deterministic environment assumption: action replay to return to $s_{\text{next}}$ requires determinism; stochastic extensions are discussed but not evaluated.
LLM API cost: MAR and global frontier analysis require multiple LLM calls per iteration; cost scales with frontier size and trajectory count.
Fixed hyperparameters: $n=3$ and $k=5$ are not adapted per game difficulty or budget.
No improvement on some games: Pentari and Detective show no gain over strong LLM baselines, suggesting GLoW's advantage is specific to games with deep bottleneck structures.
Benchmark scope: evaluation limited to text-based Jericho games; generalization to embodied or GUI environments is undemonstrated.

Relevance to Agentic AI / LLM Agents¶

GLoW directly addresses one of the most persistent weaknesses of LLM agents—inability to learn from sustained, sparse-reward exploration—by formalizing a structured memory architecture that decouples global strategic knowledge (what regions to revisit) from local tactical learning (which actions yield progress). The MAR mechanism offers a practical, training-free route to advantage estimation that is directly applicable to any agentic loop where the LLM must choose among actions with delayed feedback. The 100–800× sample efficiency improvement over RL baselines is significant for real-world deployments where environment interaction is expensive. The work establishes a blueprint for integrating RL concepts (Go-Explore, advantage functions, UCB) into LLM agent design without gradient-based training.