Skip to content

Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition

🕒 Published (v1): 2026-06-30 02:34 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper distills chain-of-thought reasoning from DeepSeek-R1 into Qwen2.5-7B using a dual-agent teacher-verifier pipeline on the John O'Bryan Mathematics Competition corpus (671 problems, 2011–2025). LoRA fine-tuning with early stopping at iteration 200 raises student accuracy from 64.67% to 69.43% ± 0.17%, with cross-domain transfer to MATH-500 (73.1% ± 0.18%). A six-level token budget experiment shows accuracy drops monotonically from 69.43% at unrestricted generation (mean 220 words) to 41.9% at a 100-token cap (mean 31.2 words).

Problem

Most CoT distillation studies use saturated benchmarks (MATH-500, GSM8K) with elevated pretraining contamination risk. There is no systematic study of how distillation performs on a domain-specific, expert-curated, low-contamination competition corpus, nor a controlled quantification of how token budget constraints at inference time degrade multi-step mathematical reasoning.

Method

Data construction: 671 problems from 15 years of the John O'Bryan competition are chronologically split (train: 2011–2021, val: 2022–2023, test: 2024–2025). A dual-agent DeepSeek-R1 pipeline generates and verifies CoT traces: Agent 1 (Solver) produces step-by-step reasoning with a delimited final answer; Agent 2 (Verifier) checks against the ground-truth answer and returns a CORRECT/INCORRECT verdict. Only verified-correct traces (594/651 on the competition set) are retained.

Fine-tuning: Qwen2.5-7B-Instruct (4-bit quantized) is adapted with LoRA (\(\Delta W = BA\), \(r=8\), \(\alpha=16\), dropout 0.05, 11.534M trainable parameters, 0.151% of total) on Apple Silicon via MLX. A diagnostic 1,000-iteration run identifies overfitting onset at iteration 200 (validation loss minimum 0.374); five independent 200-iteration runs with seeds {42, 123, 256, 512, 999} are then executed and evaluated separately.

Token budget experiment: The fine-tuned model is evaluated at six reasoning levels R1–R6 (unlimited, 400, 320, 180, 135, 100 tokens), with correctness judged by a separate DeepSeek-R1 instance.

Key Contributions

  • Construction of the first CoT distillation corpus from the John O'Bryan Mathematics Competition (671 problems, chronologically split to prevent leakage).
  • Dual-agent teacher-verifier harness that achieves 91.4% verified accuracy before committing traces to training data.
  • Demonstration that LoRA + early stopping yields a robust +4.76 pp improvement (std dev 0.17 pp across five seeds) on the held-out competition test set, with cross-domain transfer (+3.0 pp on MATH-500).
  • Quantification of accuracy degradation across six token budget levels, establishing a practical lower bound of ~50–100 words for multi-step competition reasoning.
  • Error-type analysis showing ~40% of failures are output formatting failures (not mathematical errors), pointing to post-processing as a tractable recovery path.

Results

  • Teacher (DeepSeek-R1): 91.40% on competition dataset; 94.0% on MATH-500.
  • Base Qwen2.5-7B: 64.67% on competition dataset; 70.1% on MATH-500.
  • Fine-tuned (200-iter, 5-run mean): 69.43% ± 0.17% on competition dataset; 73.1% ± 0.18% on MATH-500.
  • Overfitted model (1,000-iter): only 67.59% — worse than early-stopped model.
  • Section breakdown (fine-tuned): Freshman/Sophomore 70.0%, Junior/Senior 71.0%, Two-Person Speed 64.3%.
  • MATH-500 subject peaks: Algebra 93.5%, Number Theory 82.3%; weakest: Intermediate Algebra 52.1%, Geometry 58.5%.
  • Token budget: R1 (unlimited) 69.43% → R2 (400 tok) 61.9% → R3 (320) 60.6% → R4 (180) 51.1% → R5 (135) 43.4% → R6 (100) 41.9%.
  • Two-Person Speed section loses 34.5 pp from R2 to R6 vs. ~15–19 pp in other sections.

Limitations

  • Corpus is small (594 training traces), from a single North American institution — biases toward one pedagogical style.
  • DeepSeek-R1 serves as both teacher and verifier, introducing a stylistic dependency despite ground-truth anchoring.
  • LoRA adapters are architecture-specific (confirmed: Qwen2.5-7B adapters fail on Qwen2.5-3B due to projection dimension mismatch).
  • Training distribution skews toward algebra and number theory, leaving geometry, precalculus, and intermediate algebra weak on MATH-500.
  • Model is not deployment-ready (69.43% accuracy); requires human-in-the-loop review for educational settings.
  • No experiments with explicit training on compressed CoT to improve robustness under token constraints.

Relevance to Harnesses / Meta-Harnesses

The dual-agent teacher-verifier pipeline is a concrete instance of a multi-agent harness: one agent generates candidate reasoning traces, a second agent applies a verification oracle against an external ground truth, and only verified outputs flow into downstream training — a design directly analogous to meta-harness patterns where orchestration logic enforces quality gates before data is committed. The multi-run training protocol (diagnostic run → early-stopping calibration → five seeded production runs → fused evaluation) is itself a meta-harness workflow that wraps the raw fine-tuning tool (MLX LoRA) in a structured experimental loop. The token budget experiment further demonstrates how inference-time harness configuration (generation length caps) interacts with model capability — a concern for anyone building harnesses that constrain agent verbosity.