Think, Speak, Decide: Language-Augmented Multi-Agent Reinforcement Learning for Economic Decision-Making

🕒 Published (v1): 2025-11-17 02:09 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LAMP (Language-Augmented Multi-Agent Policy) integrates LLM-driven reasoning and inter-agent communication into a MARL framework for economic decision-making, treating natural language signals (news, peer dialogue) as first-class inputs alongside numerical observations. A three-stage Think–Speak–Decide pipeline extracts economic trends, exchanges strategic messages, and fuses language embeddings with numeric state for RL policy optimization. On the TaxAI simulator, LAMP outperforms pure MARL and LLM-only baselines in returns, robustness under economic shocks, and policy interpretability.

Problem

Standard MARL for economic decision-making ignores unstructured language signals (media narratives, peer dialogue) that are critical in real economies, assuming clean structured communication protocols. Conversely, LLM-only agents can process language but lack systematic policy optimization, making them ineffective in long-horizon, non-cooperative, multi-agent economic settings. Neither paradigm alone bridges structured numerical optimization with the semantic richness of natural language.

Method

LAMP augments a MADDPG-style MARL framework with three coupled LLM modules operating each episode step:

Think: A shared LLM news service converts global numerical indicators (Gini, GDP, welfare) into short-term shock reports (triggered when any indicator changes by threshold σ) and long-term trend analyses (at fixed checkpoints L_i). Each agent then generates private economic status assessments (κ ∈ {0,1,2}) and reasoning trajectories ψ. High-reward trajectories are stored in a dual experience pool: a short-term buffer (top-k₁ per agent) and a FAISS-indexed long-term store (top-k₂ cross-agent), with k-NN retrieval providing in-context exemplars for future LLM calls.

Speak: Each agent generates three candidate public statements via LLM, scored by a self-attention module that samples one for broadcast. Upon receiving all peers' statements V_t, each agent's reflection module Lreflect produces peer wealth-tier estimates (w^{i→j}), numeric trust scores (τ^{i→j} ∈ [0,10]), and self-reflections α^i. These update beliefs fed back into reasoning and statement selection.

Decide: A frozen text encoder (Etext) embeds private reasoning and reflections into fixed-length vectors, projected via a trainable MLP P: R^D → R^d. The fused state x_t = [O^g_t, o^{1:N_h}_t, m^{1:N_h}_t] feeds a centralized critic (MADDPG), while decentralized actors optimize deterministic policy gradients. Only the projection layer P and RL policy parameters are updated; the LLM backbone is frozen.

Key Contributions

• LAMP framework: First language-augmented MARL system explicitly modeling natural language (news + peer dialogue) as an input modality for economic decision-making.
• Think–Speak–Decide pipeline: Explicit decomposition of reasoning over trends, strategic message exchange with trust/belief updating, and language-fused RL policy execution.
• Dual experience pool: FAISS-backed long-term + short-term buffer enabling agents to retrieve and reuse high-reward reasoning trajectories as LLM in-context examples.
• Adaptive trigger scheduler: Condition-based news generation (shock threshold σ for short-term, periodic for long-term) rather than fixed-interval prompting, reducing spurious LLM calls.

Results

• vs. MADDPG (S1 stable): +63.5% average household reward, +118.8% social welfare; lower consumption (−56.8%) and labor (−60.0%), indicating efficiency gains rather than brute-force spending.
• vs. Rule-Based (S1): +12.1% reward, +12.3% social welfare; −27.9% consumption, −44.9% labor.
• vs. ReAct (strongest LLM baseline, S1): +14.5% reward, +14.8% social welfare.
• Robustness (vs. MARL baselines): +18.8% across distribution-shift scenarios (S2/S3); vs. LLM-only baselines: +59.4% robustness improvement.
• Ablations (S1):
• w/o Experience Pool: −50.9% social welfare, −50.2% stable simulation years.
• w/o Long-term reasoning: −37.7% reward, years drop from 300 → 219.
• w/o Speak: −1% welfare but +71% labor, +41% consumption (agents compensate via overwork).
• w/o Timing Scheduler (random triggers): similar welfare but −53% stable years, +82% labor.
• All LLM baselines use same backbone (Qwen2.5-72B-Instruct-INT4) and prompt budgets.

Limitations

• Evaluated exclusively on TaxAI (a single economic simulator); generalization to other domains (market pricing, labor markets) is not empirically demonstrated.
• Uses a 72B-parameter quantized LLM (Qwen2.5-72B-INT4); inference cost per step is not analyzed, making real-time or large-population deployment feasibility unclear.
• The government agent is held fixed (as in TaxAI); LAMP agents are household-only, so joint government–household language-augmented learning is not explored.
• LLM backbone is frozen during RL training; co-training the language model with the policy could yield further gains but is not attempted.
• Trust and belief representations (τ^{i→j}, w^{i→j}) are heuristic scalar scores without formal grounding in game-theoretic belief updating.

Relevance to Agentic AI / LLM Agents

LAMP demonstrates a concrete architecture for hybrid neuro-symbolic agents that combine frozen LLM reasoning with trainable RL policies—a design pattern increasingly relevant to deployed LLM agents that must act optimally over long horizons rather than producing single-turn outputs. The Think–Speak–Decide pipeline exemplifies how agentic systems can decompose language understanding, strategic communication, and decision execution into modular, composable stages. The experience pool mechanism (FAISS-backed retrieval of high-reward reasoning trajectories as in-context examples) is a practical instantiation of episodic memory for agents, directly addressing the challenge of compounding LLM inference across long episodes. This work also advances multi-agent LLM coordination beyond simple turn-taking or fixed-protocol communication, introducing trust modeling and self-reflection as primitives for adversarial or competitive multi-agent settings.