A-LAMP: Agentic LLM-Based Framework for Automated MDP Modeling and Policy Generation¶

🕒 Published (v1): 2025-12-12 04:21 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

A-LAMP is a modular multi-agent LLM framework that automates the full reinforcement learning pipeline—from free-form natural language task description to trained policy—by decomposing MDP formulation and code generation across specialized agents. It consistently outperforms single-model baselines (including GPT-4o) on both classic control and domain-specific RL tasks, with a lightweight variant built on Gemma3-27B approaching GPT-4o-level performance.

Problem¶

Converting informal task descriptions into formal MDPs and executable RL environments is manual, expertise-intensive, and fragile: a single LLM often produces syntactically valid but semantically misaligned code, or fails entirely on complex custom-environment tasks. There is no automated, end-to-end pipeline that reliably preserves semantic alignment from natural language through MDP formulation to policy training.

Method¶

A-LAMP orchestrates eight specialized LLM agents across three mirrored phases of the human expert process: 1. Abstract idea phase: A Parameter Agent extracts scope variables; an Objective Agent identifies the optimization goal; a Variable Agent distinguishes decision vs. system variables; a Constraint Agent encodes feasibility rules. 2. Formulation phase: A Modeling Agent produces a formal MDP (max_π J(π) with explicit constraints); a SAR Agent defines concrete state/action/reward structures. 3. Coding phase: An Environment Agent specifies dynamics and termination conditions; a Coding Agent writes Gym-style Python including DQN training loops; a Code Executor runs the code and feeds runtime errors back for automated debugging.

An error correction module attaches to extraction agents: each agent self-scores its output confidence, re-examines if below threshold, and optionally issues a clarification request. Agents run in dependency order (parameters → objectives → variables → constraints → modeling → coding), and all intermediate outputs are human-readable (JSON schemas, LaTeX equations, Python code).

Key Contributions¶

Structured multi-agent decomposition that narrows each agent's reasoning burden, enabling smaller LLMs to exceed single-LLM baselines.
End-to-end semantic alignment from NL description to trained policy, with intermediate outputs inspectable at every stage.
Error correction module providing self-confidence scoring and automated re-examination without full pipeline restarts.
Automated feedback loop between Coding Agent and Code Executor for runtime debugging.
Lightweight variant (Light A-LAMP on Gemma3-27B) that approaches GPT-4o performance despite a large parameter gap.

Results¶

All success rates evaluated over 20 trials per task (modeling / coding / policy generation):

Cart-pole: A-LAMP w/o EC: 1.00/0.95/0.95 vs. GPT-4o: 1.00/0.75/0.45
Mountain-car: A-LAMP w/o EC: 1.00/1.00/0.75 vs. GPT-4o: 1.00/1.00/0.40
Wireless: A-LAMP w/o EC: 0.90/0.80/0.40; A-LAMP w/ EC: 1.00/0.55/0.30 vs. GPT-4o: 0.80/0.90/0.20
Drone-delivery: A-LAMP w/o EC: 0.65/0.75/0.30; A-LAMP w/ EC: 1.00/1.00/0.45 vs. GPT-4o: 0.35/0.55/0.10
Inventory-management: A-LAMP w/o EC: 1.00/0.40/0.20; A-LAMP w/ EC: 0.80/0.95/0.45 vs. GPT-4o: 0.65/0.05/0.05
Light A-LAMP (Gemma3-27B) policy generation success rates approach GPT-4o on cart-pole (0.45 vs. 0.45) and exceed it on mountain-car (0.55 vs. 0.40).
In drone-delivery, A-LAMP eliminates the M×,C◦ spurious-coding failure mode and reduces complete failures (M×,C×) by 60%.
DQN generated by A-LAMP on the wireless task converges and matches greedy (optimal) scheduler performance at evaluation.

Limitations¶

Coding stage remains fragile: syntactic or structural mismatches can still prevent execution even when MDP modeling succeeds (e.g., inventory-management coding success rate drops to 0.40 without EC).
Policy generation success rates remain below 0.50 on the most complex custom-environment tasks even with error correction, indicating substantial remaining failure rates.
Only DQN (discrete action spaces) is tested as the RL backbone; generalization to continuous-action or model-based algorithms is unvalidated.
Error correction module requires a self-confidence threshold that is manually set; no adaptive mechanism for threshold tuning is described.
Evaluation limited to five tasks; no ablation on agent count or alternative decompositions.
No hyperparameter tuning or domain-informed priors are incorporated.

Relevance to Agentic AI / LLM Agents¶

A-LAMP exemplifies the multi-agent decomposition paradigm central to agentic AI: rather than delegating a complex end-to-end task to a single LLM, it structures the work into specialized, sequentially-dependent agents with verifiable intermediate outputs—directly addressing the coherence and reliability failures of monolithic LLM calls. The framework's self-scoring error correction and automated debugging loop are practical instantiations of agentic self-reflection and tool-use patterns. Its result that a lightweight 27B model orchestrated as multiple agents surpasses a much larger single model has direct implications for agent efficiency research. For those tracking LLM agents, A-LAMP demonstrates a concrete domain—RL environment and policy synthesis—where structured multi-agent pipelines provide measurable, reproducible gains over state-of-the-art single-model baselines.