Skip to content

P1: Mastering Physics Olympiads with Reinforcement Learning

🕒 Published (v1): 2025-11-17 17:18 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

P1 is a family of open-source physics reasoning LLMs trained purely via reinforcement learning (RL), achieving Gold-medal performance on IPhO 2025 (first open-source model to do so). The system pairs a multi-stage RL post-training framework with PhysicsMinions, an agentic test-time inference harness enabling iterative self-verification. Together, P1-235B-A22B + PhysicsMinions ranks No.1 on both IPhO 2025 and the HiPhO 13-competition suite.

Problem

LLMs capable of science-grade physics reasoning—binding symbolic manipulation to empirical law—remain scarce among open-source models. Existing RL post-training approaches suffer from reward sparsity, entropy collapse, training stagnation, and train-inference engine mismatch, limiting sustained improvement. Additionally, no prior open-source model had reached Gold-medal level on international physics Olympiads.

Method

Train-time. P1 is post-trained with Group Sequence Policy Optimization (GSPO), which computes length-normalized sequence-level importance ratios:

\[s_i(\theta) = \exp\!\left(\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\log\frac{\pi_\theta(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_\text{old}}(y_{i,t}|x,y_{i,<t})}\right)\]

Two adaptive mechanisms sustain learnability: (1) Preliminary pass-rate filtering retains only problems with \(0 < \text{pass}(q) \le 0.7\) under pass@88, eliminating zero-advantage and entropy-collapsing samples; (2) Adaptive exploration space expansion progressively increases rollout group size \(G\) and maximum generation length as training advances. Training stability is maintained via Truncated Importance Sampling (TIS) to correct for rollout–training engine mismatch.

Reward. Binary correctness per sub-answer, averaged: \(R = \frac{1}{N}\sum_{i=1}^N r_i\). A hybrid verifier combines SymPy-based symbolic checking (rule-based) with an LLM judge (Qwen3-30B-A3B-Instruct, used only for validation).

Test-time. PhysicsMinions is a multi-agent agentic framework providing iterative correction and self-verification at inference, extending effective reasoning depth without retraining.

Data. A curated corpus of 5,065 text-only physics Olympiad problems (4,126 from competitions, 939 from textbooks) with rule-verifiable answers. Quality control uses a three-model consensus pipeline (Gemini-2.5-Flash, Claude-3.7-Sonnet, GPT-4o) for answer cross-validation, filtering from 6,516 to 5,065 items.

Key Contributions

  • P1-235B-A22B: first open-source model to achieve Gold-medal performance at IPhO 2025; wins 12/13 golds on HiPhO.
  • Multi-stage RL post-training framework with adaptive learnability adjustment (pass-rate filtering + dynamic group/window expansion) and TIS-based training stabilization.
  • PhysicsMinions agentic inference harness enabling multi-turn self-verification at test time.
  • Full open-source ecosystem: models, training code, evaluation benchmark (HiPhO), and inference framework.
  • Demonstrated transferability: physics RL post-training improves math, coding, and general reasoning benchmarks.

Results

  • IPhO 2025 theory score: P1-235B-A22B scores 23.2 (Gold), ranking 3rd overall behind Gemini-2.5-Pro and GPT-5; P1-30B-A3B scores 21.2 (Silver, rank 8/35).
  • P1-235B-A22B + PhysicsMinions: No.1 on IPhO 2025 and highest average across all 13 HiPhO competitions.
  • HiPhO suite: 12 Gold + 1 Silver out of 13 competitions for P1-235B-A22B.
  • Generalizability: P1-30B-A3B significantly outperforms its base (Qwen3-30B-A3B-Thinking-2507) across 7 math, coding, and general reasoning benchmarks.

Limitations

  • Dataset is small (5,065 problems) and restricted to text-only, diagram-free problems; multi-modal physics reasoning is excluded.
  • Evaluation is confined to Olympiad-style closed-form problems; open-ended or proof-based physics reasoning is filtered out and not addressed.
  • PhysicsMinions details are largely deferred to a companion paper (Yu et al., 2025b); the agentic framework is not fully described in this report.
  • Training was performed on top of already-strong thinking base models (Qwen3); the contribution of the base model vs. RL post-training is not fully disentangled.
  • The model-based verifier is used only at validation time, not training; failure modes of the rule-based verifier on complex symbolic physics answers are not quantified.

Relevance to Harnesses / Meta-Harnesses

PhysicsMinions exemplifies a test-time meta-harness: it wraps a base reasoning model in a structured multi-agent loop of generation, critique, and iterative correction—demonstrating that an orchestration layer, not additional training, can push a model from Gold to No.1. The multi-stage data annotation pipeline (OCR → model-assisted parsing → three-model consensus QA → LLM audit) is itself a multi-model harness for dataset construction, directly informing how to build high-quality RLVR corpora. The adaptive learnability mechanisms (dynamic group size, generation window) constitute a curriculum meta-harness over the RL training loop, controlling what the underlying optimizer sees at each stage. Together, these components show that domain-specialist performance at frontier level now requires co-designing training harnesses, data harnesses, and inference harnesses as an integrated system.