Skip to content

Mastering Olympiad-Level Physics with Artificial Intelligence

🕒 Published (v1): 2025-11-13 17:20 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LOCA (LOgical Chain Augmentation) is an agent framework that decomposes physics problem-solving into serialized atomic Principle-Derivation tuples verified through an iterative augment-review loop. Applied to the 2025 Chinese Physics Olympiad theory section, LOCA achieves 313/320 (2.2% error rate) with Gemini 2.5 Pro, surpassing the top human score of 204 and all baseline agent methods.

Problem

LLMs solving advanced physics problems suffer from hallucinations and implicit logical leaps: derivation steps conflate the invocation of a physical law with its application, making errors hard to localize and correct. Existing iterative methods (self-refine, MAD, PSN) plateau at 88–94% scores because holistic review misses step-level faults in long reasoning chains.

Method

LOCA operates as a three-module pipeline with an iterative outer loop:

  1. Problem Interpretation Agent — converts the raw problem \(Q_{\text{raw}}\) into a structured physical description \(Q_{\text{struct}}\) (variables, constraints, boundary conditions, goals) that persists as context for all downstream steps.

  2. Logical Chain Augmentation Agent — transforms a raw draft \(S_{\text{raw}} = (s_1, \ldots, s_n)\) into an augmented chain \(S_{\text{aug}} = ((P_1, D_1), \ldots, (P_m, D_m))\) via two simultaneous operations:

  3. Chain Completion: decomposes non-atomic steps \(s_i: C_{i-1} \to C_i\) that hide an intermediate \(C_{\text{int}}\) into a sub-sequence \((s'_{i,1}, \ldots, s'_{i,k})\) with \(k > 1\).

  4. Structured Decomposition: each atomic step becomes a tuple \((P_j, D_j)\) where \(P_j \in L \cup M \cup C\) (physical law, mathematical identity, or problem constraint) and \(D_j = D(C_{j-1}, P_j)\) is its application; context evolves as \(C_j = C_{j-1} \cup P_j \cup D_j\).

  5. Atomic and Sequential Review — two specialized sub-agents (\(R_P\), \(R_D\)) traverse \(S_{\text{aug}}\) sequentially; step \(s_j\) passes only if both agree it is correct given accepted context \(C_{j-1}\). Each step yields \((v_j, f_j) = R(s_j \mid C_{j-1})\); the global verdict is \(V = \bigwedge_{j=1}^{m} v_j\). Failures compile into feedback \(F = \bigcup_{v_j=0} f_j\) that re-enters the Augmentation Agent. The loop terminates when consecutive correct verdicts reach \(N_{\text{corr}}^{(\max)}\) or cumulative wrong verdicts reach \(N_{\text{wrg}}^{(\max)}\).

Key Contributions

  • Formal \((P, D)\) atomic step representation that separates conceptual errors (wrong principle) from execution errors (wrong derivation), enabling targeted correction.
  • Sequential, step-conditioned review that decouples error propagation: \(s_j\) is judged assuming \(C_{j-1}\) is correct, so a local error does not contaminate the review of other steps.
  • Confidence-threshold loop termination using dual counters (\(n_{\text{corr}}\), \(n_{\text{wrg}}\)) to handle LLM stochasticity.
  • Near-perfect benchmark scores on two unseen, post-hoc evaluations (CPhO 2025 and IPhO 2025) that rule out data contamination.

Results

  • CPhO 2025 theory (320 pts): LOCA + Gemini 2.5 Pro scores 313/320 (2.2% error rate) vs. top human 204, vs. direct prompting 282 (12% error), vs. best non-LOCA baseline (Few-Shot CoT) 302 (5.6%).
  • LOCA consistently improves all five tested base models (Gemini 2.5 Pro, GPT-5, Gemini 2.5 Flash, o3, Doubao Seed 1.6) over direct prompting; improvements range from ~31 pts (Gemini 2.5 Pro) to ~35 pts (Doubao Seed 1.6).
  • Against domain-specific baseline PSN: LOCA 313 vs. PSN 292 (8.8% error).
  • IPhO 2025 (30 pts): LOCA + Gemini 2.5 Pro scores 28.6/30 vs. 26.4/30 for direct prompting under identical conditions.
  • LOCA solves at least two sub-problems that all baseline methods fail.

Limitations

  • Ablation studies and per-component analysis are deferred to Supplemental Materials; the main text does not quantify the contribution of each module individually.
  • IPhO figures were replaced with textual descriptions to isolate theoretical reasoning, meaning vision understanding is not tested in that setting.
  • Evaluation is confined to two competitions; generalization to open-ended research-level physics (non-Olympiad format) is asserted but not demonstrated.
  • The augment-review loop's computational cost (multiple LLM passes per problem) is not analyzed; scalability to larger problem sets is unstated.
  • Termination thresholds \(N_{\text{corr}}^{(\max)}\) and \(N_{\text{wrg}}^{(\max)}\) are not reported in the main text.

Relevance to Harnesses / Meta-Harnesses

LOCA is a domain-specialized reasoning harness whose architecture directly instantiates key meta-harness patterns: a structured multi-agent pipeline (Interpretation → Augmentation → Review) with an iterative outer loop, explicit pass/fail criteria, and feedback routing back to an upstream stage. The confidence-threshold loop termination (\(n_{\text{corr}}\), \(n_{\text{wrg}}\) counters) is a lightweight harness-level control mechanism analogous to retry-with-feedback loops in general agentic harnesses. The formal separation of the augmentation pass from the review pass mirrors the generate-then-verify decomposition common in scientific reasoning harnesses. For researchers building meta-harnesses, LOCA demonstrates that hard-coded sequential traversal (rather than holistic LLM judgment) is a high-leverage harness design choice that yields large accuracy gains even over strong iterative baselines like Self-Refine.