Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DEDA-FP is a deep reinforcement learning algorithm for solving non-stationary Mean Field Games (MFGs) in continuous state-action spaces, the first method capable of learning both the Nash equilibrium policy and the equilibrium mean field distribution for this general setting. It combines Fictitious Play with SAC/PPO for best-response computation, supervised learning for average policy representation, and a time-conditioned Conditional Normalizing Flow (CNF) for density estimation.

Problem¶

Existing RL methods for MFGs are restricted to either finite/discrete state-action spaces or stationary population distributions. No prior RL algorithm could learn Nash equilibria for non-stationary MFGs with continuous spaces and general dynamics/rewards, particularly when rewards depend locally on the population density (e.g., congestion terms requiring pointwise density evaluation rather than just sampling).

Method¶

DEDA-FP iterates three steps within a Fictitious Play loop: 1. Best-response computation: SAC or PPO trains a policy against N−1 agents using the current average policy network. 2. Average policy learning: A supervised MLP minimizes negative log-likelihood over a replay buffer of (t, s, a) triples from all prior best-response policies, outputting a Gaussian policy conditioned on time and state. 3. Mean field distribution learning: A time-conditioned Neural Spline Flow (CNF) is trained via MLE on trajectories generated by the current average policy, enabling both sampling and pointwise density evaluation at any (x, t).

The CNF is initialized from the previous iteration, enabling warm-starting. Convergence is analyzed via Theorem 1, which bounds exploitability as a sum of accumulated best-response error, supervised learning error, and CNF approximation error.

Key Contributions¶

First RL algorithm to solve non-stationary continuous-space MFGs with general dynamics and rewards, including local density-dependent rewards.
DEDA-FP jointly learns the Nash equilibrium policy and the equilibrium mean field flow (not just one or the other).
Time-conditioned Conditional Normalizing Flow enables exact density queries, eliminating the need for Gaussian convolution approximations of µ(x).
Formal error propagation bound (Theorem 1) connecting exploitability to three sources of approximation error.
10× sampling efficiency improvement over trajectory-ensemble baselines (1.52±0.23 s vs. 16.76±1.54 s to generate 5000 trajectories at T=20).

Results¶

Beach Bar (1D, T=10): DEDA-FP achieves smoother equilibrium distribution concentrated around the bar location; comparable exploitability convergence to Algo. 1 and Algo. 2.
Linear-Quadratic (1D, T=20): All three algorithms drive exploitability to near-zero within several FP iterations; DEDA-FP matches convergence without degradation.
4-rooms exploration (2D, T=20, entropy-maximization reward): DEDA-FP is the only algorithm capable of directly evaluating local density in the reward; with a fixed 10s time budget it can simulate 10× more agents than Algo. 2; sampling time 1.52±0.23 s vs. 15.14±1.19 s (Algo. 2) and 16.76±1.54 s (Algo. 1).
All results are means ± std over 4 independent runs.

Limitations¶

No complete theoretical convergence analysis for the neural network training components; Theorem 1 bounds assume idealized function approximation.
Evaluation relies on approximate exploitability (best-response approximated by RL), which is environment-dependent and not a certified Nash gap.
Extensions to multi-population MFGs, graphon games, and MFGs with common noise are left to future work.
Real-world applications not demonstrated; all experiments are synthetic benchmarks.
Computational scaling to higher-dimensional state spaces (d > 2) is not studied.

Relevance to Agentic AI / LLM Agents¶

MFGs provide a mathematically rigorous foundation for large-population multi-agent systems where individual agents respond optimally to collective behavior — directly relevant to designing scalable agent populations that reach stable equilibria without explicit coordination. DEDA-FP's ability to handle continuous spaces and non-stationary dynamics makes it applicable to realistic agentic settings (e.g., fleets of autonomous agents in dynamic environments, resource-contention scenarios). The hybrid architecture — RL for best-response, supervised learning for policy distillation, generative models for belief/distribution representation — mirrors emerging patterns in learned multi-agent coordination. The >10× inference speedup via normalizing flows over trajectory ensembles is practically significant for any system that must evaluate population-level state during rollout.