Skip to content

A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems

🕒 Published (v1): 2025-04-12 01:27 UTC · Source: Arxiv · Venue: TMLR · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A comprehensive survey of LLM reasoning organized along two orthogonal dimensions—regime (inference-time scaling vs. learning-to-reason) and architecture (standalone LLM vs. single/multi-agent)—unified under input/output perspectives. It covers the full spectrum from CoT prompting through RL-based training (PPO, GRPO), to agentic workflow patterns such as generator-evaluator loops and multi-agent coordination. Milestone systems (OpenAI o1, DeepSeek-R1, Grok 3 Deep Search, Manus) are contextualized within this taxonomy.

Problem

Prior surveys addressed only one or two of the four quadrants formed by regime Ă— architecture: most covered inference scaling for standalone LLMs, leaving the intersection of learning-to-reason with agentic/multi-agent systems largely unsurveyed in a unified framework. No existing work provides a single organizing lens that spans from prompt engineering to RL training and from single-model inference to multi-agent orchestration.

Method

A 2D taxonomy partitions the literature: - Regime axis: (1) Inference-time scaling—frozen-model trajectory search guided by ORMs/PRMs; (2) Learning-to-reason—parameter updates via SFT on long-CoT, preference learning (DPO), or RL (PPO, GRPO). - Architecture axis: (1) standalone LLM; (2) single-agent (LLM + tools + environment); (3) multi-agent (agent–agent communication + coordination).

Each cell is analyzed through a unified input/output lens: at the input level, techniques construct or augment prompts/perceptions/communication; at the output level, they aggregate, rank, refine, or coordinate multiple candidate trajectories. Reasoning is formally grounded in the MDP framework: \(\tau = \langle(s_0,a_0,r_0),\ldots,(s_T,a_T,r_T)\rangle\), with the objective \(\max \mathbb{E}_{\tau \sim P(\tau|s_0,\pi)}\sum_{t=1}^{T} r_t\).

Key Contributions

  • Two-dimensional (regime Ă— architecture) taxonomy covering most existing reasoning approaches
  • Unified input/output perspective that bridges inference-scaling and learning-to-reason, and generalizes to perception/action (single-agent) and communication/coordination (multi-agent)
  • Systematic coverage of learning algorithms: SFT, DPO, PPO, GRPO, ORMs, PRMs, generative verifiers
  • Survey of agentic workflow design patterns: generator-evaluator, LLM debate, tool-augmented single agents, multi-agent coordination protocols
  • Identification of emerging trends: inference-aware learning-to-reason (\(§5.4\)), automated workflow design, domain-specific reasoning (math, code, tabular, game)
  • Discussion of open challenges: evaluation quality, training data scarcity, cost-aware training

Results

Survey paper; no new experimental results. Cited milestones and their community impact: - OpenAI o1 (09/2024): demonstrated consistent performance improvement with both train-time and test-time compute scaling - DeepSeek-R1 (01/2025): open-source model matching o1 performance via RL alone, without step-level annotations - OpenAI Deep Research and Grok 3 Deep Search (02/2025): single-agent systems combining web browsing, code execution, and multi-source aggregation - Manus: multi-agent product demonstrating agent-agent coordination at scale - Cumulative paper volume on Semantic Scholar (2022–02/2025): sharp acceleration since CoT (2022), ChatGPT (09/2022), and o1 (09/2024)

Limitations

  • Explicitly defers benchmark analysis to companion surveys (Xu et al., 2025d; Besta et al., 2025)
  • Concurrent work by Besta et al. (2025) covers reasoning structures and benchmarks more formally; this survey's agentic coverage is complementary but not cross-validated
  • Truncated source text means Sections 3–6 (the bulk of technical depth) cannot be verified from the provided excerpt
  • Agent-human interaction regimes and fully automated workflow design (Hu et al., 2025; Zhou et al., 2025a) are identified as future directions, not yet surveyed

Relevance to Harnesses / Meta-Harnesses

The survey's agentic workflow patterns—generator-evaluator loops, multi-agent coordination, tool-augmented reasoning pipelines—are the conceptual substrate from which harnesses and meta-harnesses are engineered. The MDP framing of trajectories (\(\tau\)) and the input/output decomposition map directly onto harness design: input-level prompt construction corresponds to harness prompt templating, and output-level trajectory selection/refinement corresponds to harness routing and aggregation logic. The coverage of automated workflow design (Hu et al., 2025; Zhang et al., 2024c; Zhou et al., 2025a) signals that meta-harness generation itself is becoming a first-class research problem, relevant to anyone building self-configuring or self-optimizing orchestration layers. The inference-aware learning-to-reason direction (\(§5.4\)) implies that future harnesses may need to co-design their orchestration logic with the training procedure of the underlying reasoner.