Skip to content

MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Large-scale training environment for biomedical code-centric LLM agent reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedAgentGym is a scalable, interactive training environment of 72,413 code-centric biomedical data science tasks across 129 categories, backed by isolated Docker sandboxes and multi-turn trajectory sampling. It benchmarks 29 LLMs and demonstrates that fine-tuning a 7B open-source model (Med-Copilot) with offline/online RL achieves +43%/+45% gains, reaching parity with gpt-4o while preserving privacy.

Problem

Biomedical data science requires both medical domain knowledge and programming proficiency (EHR queries, genomic analysis, ML modeling), yet no prior environment offered: (1) a comprehensive, executable, code-centric benchmark spanning the full biomedical data science lifecycle, and (2) an interactive training ground with verifiable rewards for RL-based agent fine-tuning. Existing benchmarks are single-pass, non-executable QA sets, and proprietary LLMs are prohibitive under clinical privacy constraints while open-source LLMs lag far behind.

Method

MedAgentGym formalizes each task as \((x, y^*)\): given problem description \(x\), an agent generates code \(c\) executed to produce output \(y\), verified by \(E(c, y) = \mathbb{I}(y = y^*)\). Tasks are served inside per-task Docker containers pre-installed with biomedical libraries, ensuring isolation and reproducibility. Agents operate under a POMDP scaffold (following CodeAct) with four action types: request_info, terminal, code_execution, and debugging. Error messages are translated to natural language for LLM comprehension. Trajectories—both positive and negative—are collected at scale via Ray/Joblib multi-threaded parallel rollouts. Med-Copilot is then trained on these trajectories via (a) offline RL: supervised fine-tuning on sampled trajectories, and (b) online RL: iterative policy rollout → reward → update within the live gym environment.

Key Contributions

  • 72,413 verifiable task instances across 129 categories from 12 real-world biomedical scenarios (EHR SQL, clinical calculations, bioinformatics, ML modeling, etc.), split into in-distribution (40K train) and out-of-distribution (32K) sets with n-gram (\(n=10\)) decontamination.
  • Isolated, reproducible Docker sandbox environments per task with interactive multi-turn feedback, LLM-friendly error grounding, and dynamic dependency installation.
  • Scalable trajectory collection engine (Ray + Joblib) supporting both offline and online RL fine-tuning pipelines.
  • Benchmark of 29 LLMs (6 proprietary, 23 open-source), revealing a large OSS-vs-commercial performance gap (avg 16–32 vs 37–70 overall score).
  • Med-Copilot (7B and 14B), open-source biomedical coding agents fine-tuned on MedAgentGym trajectories, competitive with gpt-4o on both in- and out-of-distribution tasks.
  • Public release of environment, benchmark, 6.7K training trajectories, and outcome verifier.

Results

  • Overall leaderboard average: gpt-4.1 tops at 70.15; best OSS baseline (Qwen3-32B) at 48.19; Med-Copilot-7B reaches 50.07 after RL training.
  • Med-Copilot-7B gains: +43.02% (offline RL) and +45.28% (online RL) over the base Qwen2.5-7B-Instruct (16.97 → ~24–25 avg, exact post-training numbers implied by the gain percentages).
  • Proprietary gap: Top OSS (DeepSeek-R1-Distill-Llama-70B, 50.07) approaches but does not exceed gpt-4o (48.75); gpt-4.1 (70.15) and gpt-o4-mini (65.67) remain substantially higher.
  • Task-specific: Structured tasks (database queries, medical calculations) yield higher SR than open-ended ML modeling tasks across all models.
  • Medical domain-specific LLMs (HuatuoGPT, MedReason, Baichuan-M) do not consistently outperform general-purpose models of similar scale on coding tasks.
  • Out-of-distribution: Med-Copilot remains competitive with gpt-4o on held-out datasets (EHR-SeqSQL, EHRCon, MIMIC-Extract, N-PowerAI).

Limitations

  • Scope restricted to code-centric tasks; knowledge-intensive medical QA (MedMCQA, MedQA) is explicitly excluded as orthogonal.
  • Open-ended ML tasks (EHRSHOT, MIMIC-Extract) reuse the same data for train and test due to absence of separate ground truth, which may inflate those scores.
  • Docker-based isolation adds infrastructure overhead; industrial deployment under strict HPC/cloud constraints may face friction.
  • Benchmark covers only tasks with verifiable ground-truth outputs; subjective clinical reasoning or free-text generation tasks are not captured.
  • Performance gap between Med-Copilot and top proprietary models (gpt-4.1 at 70.15 vs Med-Copilot-14B at ~51) remains large.

Relevance to Agentic AI / LLM Agents

MedAgentGym directly instantiates the agentic RL paradigm—POMDP formulation, multi-turn action loops, environment-grounded reward signals—in a high-stakes domain where privacy constraints make open-source agent development essential. The +43–45% gains from RL training demonstrate that interactive, executable environments with verifiable rewards are far more effective training signals than static SFT on QA pairs, generalizing the lesson from WebArena/SWE-bench to biomedical data science. The public release of trajectories, verifier, and Docker infrastructure lowers the barrier for the community to apply agentic fine-tuning pipelines (GRPO, PPO, etc.) to specialized professional domains. It also establishes a reproducible multi-task evaluation harness that the field currently lacks for coding agents in clinical settings.