Skip to content

AlphaAgentEvo: Evolution-Oriented Alpha Mining via Self-Evolving Agentic Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Self-evolving agentic RL for systematic alpha factor mining in quantitative finance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AlphaAgentEvo is a self-evolving Agentic Reinforcement Learning (ARL) framework that reformulates quantitative alpha mining from a brittle "search–backtest–restart" cycle into a continuous multi-turn evolution trajectory. It extends GRPO to a multi-turn, tool-in-the-loop setting with a hierarchical reward function, enabling a 4B-parameter LLM agent to acquire long-horizon planning and reflective reasoning. The 4B model consistently outperforms state-of-the-art closed-source LLMs (GPT-5-mini, DeepSeek-R1) and multi-agent baselines on alpha evolution benchmarks.

Problem

Existing alpha mining approaches fail to support systematic, principled evolution. Genetic Programming (GP) relies on heuristic random mutation, cannot interpret natural language, and cannot learn from failures. LLM/multi-agent approaches (e.g., AlphaAgent, GEPA) lack mechanisms for long-term planning and reflective reasoning, causing them to fall into repetitive local modifications and inefficient exploration. Single-turn RL methods (e.g., ToolRL) also fail to generalize across multi-turn trajectories. None of these methods moves beyond per-episode search toward a genuinely evolving policy.

Method

AlphaAgentEvo reformulates alpha mining as learning an evolution policy \(\pi^*\) that, given a seed alpha \(f_\text{seed} \sim D_\text{seed}\), produces evolved alphas \(\mathcal{F}_\pi(f_\text{seed})\) maximizing in-distribution performance plus \(\lambda \times\) out-of-distribution generalization subject to an AST-based structural similarity constraint \(\text{sim}(f, f_\text{seed}) \leq \delta\).

The training method is Agentic GRPO (ARL): at each turn \(t\), the policy LLM generates reasoning tokens and \(k_t\) parallel alpha proposals (tool calls) conditioned on the entire past trajectory \(\tau_{1:t-1}\); proposals are evaluated by an external backtesting tool. Only policy-generated tokens (masked by \(M_{i,t}\)) contribute to gradients. Group advantage estimation normalizes rewards within a group of \(G\) trajectories rolled out from the same seed.

A hierarchical reward \(R(\tau)\) combines five capped components multiplicatively/additively: - \(R_\text{tool}\): rewards successful tool calls, penalizes failures - \(R_\text{cons}\): penalizes alphas with AST-similarity to seed below threshold \(h_\text{low}=0.1\), preventing random drift - \(R_\text{expl}\): rewards diversity by penalizing high AST-similarity to previously proposed alphas - \(R_\text{perf}\): logarithmically scaled reward for Information Ratio improvement over seed - \(R_\text{streak}\): bonus proportional to the longest streak of monotonically improving alphas within a trajectory

\[R(\tau) = \underbrace{\min(R_\text{tool}, C_\text{tool})}_{\text{cost normalizer}} \cdot \left[\min(R_\text{cons}, C_\text{cons}) + \min(R_\text{expl}, C_\text{expl}) + \min(R_\text{perf}, C_\text{perf}) \cdot \min(R_\text{streak}, C_\text{streak})\right]\]

Training uses the expert-curated AlphaEvo500 dataset (350 train / 50 val / 100 test seed alphas); models are fine-tuned from Qwen3-1.7B and Qwen3-4B-thinking.

Key Contributions

  • First self-evolving ARL framework for quantitative alpha mining, replacing episodic "search–backtest–restart" with a continuous multi-turn evolution trajectory
  • Extension of GRPO to multi-turn, tool-in-the-loop agentic settings where only policy-emitted tokens receive gradients
  • Hierarchical reward function decomposing sparse financial backtesting feedback into dense, multi-objective signals (validity, consistency, diversity, performance, streak)
  • Empirical demonstration that a 4B ARL-trained model surpasses state-of-the-art closed-source LLMs and multi-agent baselines on alpha evolution benchmarks
  • AlphaEvo500 benchmark (500 expert-curated seed alphas) for controlled evaluation of evolution capability

Results

AlphaEvo500 (HS300 / CSI500, 2024–2025): - AlphaAgentEvo-4B: pass@3 = 0.97 / 0.93, pass@5 = 0.97 / 0.95 - vs. best baseline GEPA (DeepSeek-R1): pass@3 = 0.83 / 0.82, pass@5 = 0.87 / 0.88 - vs. GPT-5-mini: pass@3 = 0.75 / 0.73, pass@5 = 0.88 / 0.82 - AlphaAgentEvo-1.7B already surpasses GPT-5-mini on HS300

Alpha158 (external generalization test, 2023–2025): - AlphaAgentEvo-4B (bullish period): pass@5 = 0.994 vs. DeepSeek-R1 = 0.943, GPT-5-mini = 0.903 - AlphaAgentEvo-4B (bearish period): pass@5 = 0.725 vs. DeepSeek-R1 = 0.519 - GP (4 offspring): pass@3 ≈ 0.000–0.003, dramatically worse

Ablation (AlphaEvo500 pass@3): - Full model: 0.65 → removing exploration reward: 0.54 → removing consistency reward: 0.51

Diversity: AlphaAgentEvo achieves mean pairwise similarity 0.039, max 0.263 vs. DeepSeek-R1 (max 0.583) and Qwen3-4B (max 0.600), indicating less pattern over-concentration.

Limitations

  • Trained and evaluated exclusively on Chinese equity markets (HS300, CSI500); transferability to other asset classes or markets is untested
  • The AlphaEvo500 benchmark is self-constructed and not yet community-validated
  • Backtesting uses only 1 year of training data to accelerate iterations, which may underfit longer market cycles
  • The ARL training process requires an expensive external backtesting tool in the training loop, increasing computational cost relative to standard LLM fine-tuning
  • Structural similarity via AST overlap is a proxy for interpretability and does not guarantee economic interpretability
  • The paper is truncated; diversity and transferability analysis (Fig. 5 and portfolio multi-factor results) are partially missing

Relevance to Agentic AI / LLM Agents

AlphaAgentEvo is a concrete demonstration of tool-augmented multi-turn RL moving beyond single-turn RLHF paradigms: it trains an LLM agent to plan and reflect across long trajectories with real external-tool feedback, directly relevant to the emerging line of work on agentic RL (GRPO extensions, ToolRL, etc.). The hierarchical reward design—decomposing sparse, delayed feedback into dense intermediate signals—is a transferable technique for any agentic domain where final outcomes are high-variance (code execution, scientific experiments, robotics). The finding that a 4B ARL-trained model outperforms much larger closed-source reasoning models (DeepSeek-R1, GPT-5-mini) on a multi-turn planning task supports the hypothesis that domain-specific agentic RL can be more sample-efficient than scale alone. The "streak reward" and "consistency reward" mechanisms are novel handles for shaping exploration-exploitation tradeoffs in multi-turn agent training.