Skip to content

Vibe Reasoning: Eliciting Frontier AI Mathematical Capabilities -- A Case Study on IMO 2025 Problem 6

🕒 Published (v1): 2025-12-22 11:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vibe Reasoning is a human-AI collaborative paradigm built on four pillars—AI as primary reasoner, Socratic meta-prompts, agentic grounding (code execution + file-based memory), and model orchestration—designed to unlock frontier models' latent capabilities via minimal, domain-agnostic human guidance. The authors validate the approach by solving IMO 2025 Problem 6, a combinatorial tiling problem on which all autonomous AI systems publicly failed, obtaining both the correct answer (2112) and a complete mathematical proof. The human role is purely meta-cognitive: timing verifications, switching models, and pruning search space—never supplying mathematical content.

Problem

Frontier LLMs possess the mathematical knowledge needed for hard competition problems but lack the judgment to deploy it: they hallucinate with high confidence, cycle through failed proof strategies without pivoting, lose context across sessions, and cannot reliably self-evaluate. Fully autonomous AI systems failed IMO 2025 P6 entirely (0 of ~600 human solvers did too, except 6). The gap is not knowledge but orchestration and grounding.

Method

Vibe Reasoning structures human-AI interaction around four orthogonal components:

  1. AI as primary reasoner — AI generates all mathematical content (hypotheses, constructions, proofs) autonomously; humans supply no domain knowledge.
  2. Socratic meta-prompts — generic, transferable human directives ("verify with code", "try small cases", "this seems wrong") trigger AI's latent verification/reflection capabilities without revealing answers.
  3. Agentic grounding — Python execution catches hallucinations in real time (e.g., refutes wrong formula M(n)=2n−2 via exhaustive search for n≤8); file-based memory (summary.md, proof_sketch.md) maintains cross-session coherence (~106 file reads/writes in successful traces).
  4. Model orchestration — GPT-5 handles open-ended exploration and pattern discovery; Gemini 3 Pro handles rigorous proof construction; the human decides when to switch phases.

Applied to IMO P6: GPT-5 enumerated small cases, identified residue-block permutation structure, and conjectured M(k²)=k²+2k−3; Gemini 3 Pro proved the lower bound using the Fooling Set / Erdős-Szekeres / LIS-LDS construction. The method was refined across five attempts totaling >24,000 lines of interaction traces, with human prompts evolving from problem-specific hints toward fully generic meta-prompts.

Key Contributions

  • Definition and taxonomy of Vibe Reasoning, including a categorized prompt table (Verification, Specialization, Challenge, Simplification, Output, Memory).
  • First reported complete solution (answer + proof) of IMO 2025 P6 by any AI-assisted system.
  • Empirical analysis of model complementarity: GPT-5 strong on exploration/code, weak on formal proof; Gemini 3 Pro opposite.
  • Root-cause analysis of autonomous AI failure: knowledge-application gap, overconfidence / verification blindness, circular proof cycling, context loss.
  • Longitudinal trace analysis showing convergence from domain-specific to generic meta-prompts across five iterative attempts.

Results

  • IMO 2025 P6 solved: correct answer M(2025) = 2112 and complete lower-bound proof (Fooling Set via ErdÅ‘s-Szekeres, Adaptive Orthogonal Fanning construction).
  • Computational verification of Fooling Set construction on random permutations: 98% success rate at n=10, 100% for residue permutation at n=25.
  • Successful answer trace used ~106 file references; proof trace ~43; early failed traces used near zero (correlated with context loss and failure).
  • No other quantitative benchmark comparisons reported; paper is a single-problem case study.

Limitations

  • Single problem case study—generality to other competition or domain problems is asserted but not yet evaluated.
  • Human judgment for phase transitions, model selection, and search-space pruning is not yet automatable; the authors acknowledge correlated-error risk in AI-verifying-AI.
  • Human prompts still required non-trivial meta-cognitive insight (e.g., recognizing 2025=45² as a perfect square warranting special focus).
  • Work in progress; automated framework and broader benchmark evaluation are described as future work only.
  • Interaction traces are condensed/edited in the paper; reproducibility from the main text alone is limited.

Relevance to Harnesses / Meta-Harnesses

Vibe Reasoning's "agentic grounding" pillar is essentially a lightweight harness: Python subprocess execution + persistent file-based scratch memory forms an external scaffolding layer that compensates for LLM context limits and hallucination. The model orchestration pillar—routing tasks between GPT-5 and Gemini 3 Pro based on task phase—is a minimal meta-harness pattern (task decomposition → model dispatch → result handoff). The paper's future work explicitly targets codifying meta-prompt patterns into reusable templates and decision rules, converging directly on what a meta-harness would automate. This makes it a concrete, worked example of an emerging meta-harness design where the human currently implements the routing logic that a harness would eventually encode.