Skip to content

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

🕒 Published (v1): 2025-11-24 18:35 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DR Tulu introduces Reinforcement Learning with Evolving Rubrics (RLER), a training method that co-evolves per-instance evaluation rubrics with the policy model during RL to provide grounded, discriminative feedback for long-form deep research. DR Tulu-8B, trained on Qwen3-8B with SFT then RLER, becomes the first fully open model trained end-to-end for open-ended long-form deep research, outperforming all open baselines and matching proprietary systems at ~1000Ă— lower cost.

Problem

Existing open deep research agents are either training-free or trained via RLVR on short-form QA, which fails to transfer to realistic long-form research tasks. Long-form deep research requires rewards that are knowledge-intensive, under-specified, and vulnerable to reward hacking—problems that static, predefined rubrics cannot handle because they lack grounding in external knowledge and cannot adapt as the policy explores new information.

Method

RLER maintains a per-instance rubric buffer initialized by searching the web for each training prompt and generating persistent rubrics from retrieved documents. At each GRPO rollout step, a rubric LM (GPT-4.1) receives the prompt, all sampled rollouts (including search traces and final answers), and the existing rubric buffer, then generates two types of evolving rubrics: (1) positive rubrics capturing strengths and newly discovered facts; (2) negative rubrics flagging reward-hacking patterns (e.g., verbatim copying to inflate citation precision). Rubric buffer management prunes rubrics with zero variance in rollout scores and retains the top-K by standard deviation. Auxiliary rewards for formatting, search use, and snippet-level citation quality supplement the rubric reward. Training uses a customized GRPO with asynchronous tool calls (dispatching tool requests immediately during rollout generation), token-level loss, and tool-output masking. The model is first cold-started with SFT on 16K GPT-5-distilled trajectories mixing long-form and short-form tasks, then RL-trained using ~9K long-form prompts from SearchArena, OpenScholar, and RaR.

Key Contributions

  • RLER: RL training with instance-specific rubrics that co-evolve with the policy, grounded in external knowledge from search and contrastive across sampled rollouts.
  • DR Tulu-8B: First fully open 8B model trained end-to-end for open-ended long-form deep research, with snippet-level citations.
  • dr-agent-lib: Open agent library with MCP-based unified search/browse backend, async process locking, and composable prompt layer.
  • GeneticDiseasesQA: Expert-curated clinical deep research benchmark of 47 questions over 24 pathogenic gene variants requiring multi-source synthesis.
  • Full open release: model weights, SFT/RL training data, evolving rubrics, evaluation suite, and asynchronous RL infrastructure.

Results

  • Average across 4 long-form benchmarks (SQAv2, HealthBench, ResearchQA, DeepResearchBench): DR Tulu-8B (RL) scores 65.6, vs. best open baseline Tongyi DR-30B at 50.0 (+15.6 pts) and OpenAI Deep Research at 64.9 (+0.7 pts).
  • SQAv2: 88.3 (DR Tulu-8B RL) vs. 87.7 (Ai2 ScholarQA w/ Claude Sonnet); beats all open models including Tongyi DR-30B (46.5).
  • HealthBench: 52.8 vs. Tongyi DR-30B 46.2 vs. OpenAI DR 53.8.
  • Cost: $0.0019/query (including LM inference) vs. OpenAI DR $1.80/query (~1000Ă—) and Tongyi DR $0.03/query.
  • RLER vs. static rubrics only: up to 2-point average gain that widens over training.
  • Search-based rubrics vs. closed-book rubrics: search-based rubrics score 56.7 avg vs. closed-book 55.6 after 500 RL steps.
  • Open judge (Qwen3-8B as judge/generator): +4.4 avg points over SFT at 1000 steps vs. +5.7 for GPT-4.1 judge (1.3-pt gap).
  • Short-form QA (SimpleQA, 2Wiki, WebWalkerQA): DR Tulu-8B RL averages 61.3 despite RL training exclusively on long-form data.
  • GeneticDiseasesQA: DR Tulu-8B outperforms Ai2 ScholarQA on Final Answer correctness; competitive with GPT-5 + Search on evidence metrics.
  • RL training used 27,000 H100 GPU hours on 16 GPUs.

Limitations

  • RLER requires a capable rubric-generating LM (GPT-4.1 in the main run); the open-judge variant (Qwen3-8B) lags by 1.3 points, with context-length constraints preventing full trajectory input to the rubric generator.
  • Training cost is substantial (27,000 GPU hours), limiting accessibility for most researchers.
  • Evaluation reuses leaderboard numbers for many baselines rather than rerunning them, introducing potential protocol inconsistencies.
  • GeneticDiseasesQA covers only 47 questions / 24 gene variants; generalization to broader clinical genetics tasks is unconfirmed.
  • The rubric buffer management heuristic (select by reward variance) may drop rubrics capturing rare but critical quality dimensions.
  • Training/inference tool mismatch (Crawl4AI during training vs. Jina during inference) is not fully characterized beyond a single appendix ablation.
  • The system can still generate plausible but incorrect claims; medical/scientific deployment requires careful human oversight.

Relevance to Agentic AI / LLM Agents

RLER directly addresses a core open problem in training long-horizon tool-use agents: how to produce reliable reward signals when task quality is under-specified and depends on knowledge external to the model. The co-evolving rubric mechanism is a novel alternative to static verifiable rewards (RLVR) that scales to open-ended, multi-step agentic trajectories—bridging the gap between short-form RL and realistic research agent training. DR Tulu's fully open release (infrastructure, data, rubrics) lowers the barrier to end-to-end RL training of search agents, providing a reproducible baseline for the community. The adaptive tool-selection behavior (automatically switching between paper search and web search by task type) demonstrates emergent agentic specialization from RL, relevant to anyone building multi-tool agents trained to match search strategy to information need.