Skip to content

Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization

🕒 Published (v1): 2025-12-31 03:59 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper augments multi-agent LLM workflows with MARL: it casts a team of role-specialized LLM agents as a Dec-POMDP, trains them with centralized critic + Group Relative Policy Optimization (GRPO), and executes decentrally. On collaborative writing and coding benchmarks, the approach achieves 3× throughput and higher quality with fewer tokens than prompt-only multi-agent baselines.

Problem

LLM-based multi-agent systems (AutoGen, MetaGPT) rely on fixed prompting heuristics and hand-written coordination rules with no unified learning signal. They cannot optimize long-horizon team performance under partial observability, leading to redundant chatter, poor credit assignment, and inefficient coordination.

Method

The framework models a team of LLM agents (planner, writer, reviewer, coder, tester) as a Dec-POMDP with structured action primitives (plan, draft, integrate, lint, test, repair, finalize) and a shared/private observation split. Training uses CTDE: a centralized critic sees full transcripts and tool traces to learn coordination patterns; at inference each agent operates on a bounded local observation (artifact slice + summary rail + role scratchpad).

Policy optimization uses GRPO — a group-relative baseline that credits each agent's marginal contribution relative to what the team would achieve without that move. The joint reward combines four normalized signals: task quality (style/structure or test pass rate), processing speed (batch-normalized), coordination efficiency (penalty for redundant turns and over-long messages), and compliance/safety. A shared experience buffer stores trajectories and self-critiques; a curriculum grows episode length and interleaves task types.

Key Contributions

  • Dec-POMDP formulation of multi-role LLM agent collaboration with structured action primitives and explicit partial observability
  • GRPO applied to LLM teams: leave-one-out group baseline for stable credit assignment across open-ended language actions
  • Four-component joint reward that balances quality, speed, coordination cost, and safety, with per-turn audit fragments
  • Demonstration that CTDE + group-relative optimization yields Pareto improvement over prompting-only multi-agent baselines on quality, speed, and token cost simultaneously

Results

  • Proposed (GRPO) vs. Single LLM vs. AutoGen Team on 150 writing + 120 coding tasks:
  • Processing speed: 3.0× vs. 1.80× (AutoGen) vs. 1.00× (Single)
  • Writing structure/style: 98.7% vs. 94.2% vs. 90.1%
  • Coding unit-test pass rate: 74.6% vs. 68.1% vs. 61.3%
  • Message turns: 8.6 vs. 11.3 vs. 14.2
  • 20–35% fewer message turns and ~18–22% fewer tokens vs. AutoGen at equal or higher quality
  • Wall-clock reduction of ~60–70% vs. Single LLM, ~40–50% vs. AutoGen
  • Ablations: removing group baseline drops style to 96.9%; removing coordination cost drops to 96.1%; local-only reward drops to 94.0%
  • Failure modes (over-planning, review repetition, late testing) reduced by 41%, 36%, 29% respectively under GRPO

Limitations

  • Evaluation is narrow: 150 writing prompts and 120 coding problems; no real-world deployment or large-codebase tasks
  • Style/structure quality metrics retain subjectivity even with rubrics and blind review
  • Long documents or large codebases can overflow observation slices, degrading credit assignment
  • Noisy tool outputs complicate reward attribution
  • With unlimited token budgets, conversational baselines close part of the gap — advantage is tied to budget-constrained settings
  • No comparison against more recent RL-trained LLM agent systems (e.g., RLHF-aligned tool-use agents)

Relevance to Harnesses / Meta-Harnesses

This paper operationalizes a training harness for multi-agent LLM pipelines: the centralized critic, shared experience buffer, curriculum scheduler, budget cap, and coach/monitor together constitute a meta-level control layer that governs how sub-agents are trained and supervised — structurally analogous to a meta-harness. The CTDE separation (global training signal → local execution) mirrors the pattern of harnesses that orchestrate agents at training time but let them run independently at inference. The explicit audit trail (per-turn action logs, tool receipts, decision rationales) is a design pattern directly applicable to harness observability and debugging infrastructure.