Extending NGU to Multi-Agent RL: A Preliminary Study¶

🕒 Published (v1): 2025-12-01 06:24 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper extends the Never Give Up (NGU) single-agent exploration algorithm to cooperative multi-agent reinforcement learning (MARL) and evaluates three design choices—shared replay buffers, shared novelty signals, and heterogeneous β values—on a sparse-reward pursuit–evasion task. Multi-NGU with a shared replay buffer achieves the best performance and stability, outperforming a multi-agent DQN baseline. The work is preliminary, scoped to one environment, and identifies shared experience as the dominant contributor to gains.

Problem¶

Sparse-reward MARL is compounded by credit assignment, non-stationarity, and the need for coordinated exploration. Existing intrinsic-motivation methods for MARL (EMC, MACE, LIIR) introduce architectural complexity and may not generalize. The NGU algorithm, highly effective for single-agent sparse-reward tasks, has not been studied in the multi-agent setting.

Method¶

The authors strip NGU to its core—an embedding network trained with an inverse dynamics model, episodic memory, and k-nearest-neighbor intrinsic reward—omitting RND and UVFA for computational tractability. Each agent maintains its own Q-network, embedding network, and episodic memory; intrinsic reward is r_intrinsic = f(φ(s_{t+1}), M_i) (k-NN distance in embedding space), combined with extrinsic reward as r_total = r_extrinsic + β·r_intrinsic. Three design variants are tested: (1) shared vs. per-agent replay buffer, (2) shared novelty (state treated as non-novel once visited by k different agents, using cosine similarity), and (3) heterogeneous β values across agents (e.g., {0.1, 0.2, 0.4}). Evaluation is on simple_tag_v3 (PettingZoo), 200k timesteps, 15 independent runs.

Key Contributions¶

Direct extension of NGU's core exploration mechanism to cooperative MARL without RND or UVFA
Empirical comparison of three multi-agent design choices: replay sharing, novelty sharing, heterogeneous β
Evidence that shared replay buffer is the primary driver of Multi-NGU's gains over Multi-DQN
Identification that shared novelty at k=1 performs comparably to standard Multi-NGU, but degrades for k>1
Finding that heterogeneous β does not improve over a uniform small β=0.1

Results¶

Multi-NGU (shared buffer) achieves substantially higher average returns and more stable learning curves than Multi-DQN (shared buffer), with tighter 95% CI across 15 runs
Without shared buffer: Multi-NGU outperforms Multi-DQN but gains are moderate; novelty sharing and heterogeneous β still exceed baseline
With shared buffer: standard Multi-NGU is strongest; shared novelty (k=1) is comparable but standard Multi-NGU surpasses it in long-term returns; heterogeneous β trails
k>1 in novelty sharing degrades performance (k=1 > k=2 > k=3); heterogeneous β variants {0.1,0.2,0.4} and {0.2,0.4,0.6} both underperform homogeneous β=0.1
No absolute return numbers reported; results are relative comparisons via smoothed learning curves

Limitations¶

Evaluated on a single environment (simple_tag_v3) with one base algorithm (DQN)
No comparison to established MARL baselines beyond Multi-DQN (no QMIX, MADDPG, MAPPO, SMAC)
RND and UVFA excluded, so results do not reflect full NGU capability
Some ablation conditions run with fewer seeds (5–10) due to computational constraints
Evader follows a fixed heuristic policy, limiting task diversity
No large-scale or competitive multi-agent evaluation

Relevance to Agentic AI / LLM Agents¶

This work addresses coordinated exploration in multi-agent systems under sparse feedback—a fundamental challenge shared by LLM-based agent pipelines where reward signals are delayed or infrequent. The finding that shared experience (replay buffer) dominates over shared novelty signals is directly relevant to designing memory and experience-sharing mechanisms in multi-agent LLM frameworks. The analysis of heterogeneous exploration strategies (varying β) informs how to balance exploration/exploitation roles across specialized agents in a team. While the setting is tabular MARL rather than language agents, the design principles (what to share, when novelty helps, how to tune intrinsic motivation) transfer to any multi-agent architecture facing sparse environmental feedback.