Skip to content

StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; StepORLM: self-evolving generative process supervision for OR language models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

StepORLM is a self-evolving training framework for LLMs solving Operations Research (OR) problems, pairing a policy model with a generative process reward model (GenPRM) that co-evolve through a dual-feedback loop. An 8B-parameter model trained under this framework achieves state-of-the-art across six OR benchmarks, outperforming much larger generalist models (DeepSeek-V3 671B, GPT-4o) and specialized baselines. The co-evolved GenPRM also serves as a transferable inference-time verifier that boosts other OR LLMs.

Problem

Existing RL paradigms for training LLMs on OR tasks face two core failures: (1) outcome reward suffers from the credit assignment problem—a correct final answer can arise from flawed intermediate reasoning, reinforcing bad steps; (2) discriminative process reward models (PRMs) are myopic, scoring steps in isolation without capturing cross-step dependencies (e.g., a variable definition whose correctness only becomes apparent after subsequent constraints are introduced). OR modeling demands holistic, trajectory-level supervision because step validity is inherently interdependent.

Method

StepORLM trains in two stages:

Stage 1 — Warm-Up SFT. A teacher LLM (GPT-4o) generates 50K diverse (question, step-by-step reasoning trajectory) pairs. Each solution is executed against an external solver (COPT); failures trigger an automated self-refinement loop. The verified corpus trains an initial policy \(\pi_0\) via supervised fine-tuning.

Stage 2 — Iterative Co-Evolution. Starting from \(\pi_0\) and a base LLM as initial GenPRM \(\rho_{\phi_0}\), the framework iterates over a hard subset \(D_{\text{hard}}\) (13K problems where majority voting fails unanimously): 1. Trajectory collection: The current policy \(\pi_\theta\) generates \(k=4\) candidate trajectories per problem. 2. Dual-source evaluation: (a) An external solver provides binary outcome labels; (b) GenPRM performs chain-of-thought reasoning over the complete trajectory and outputs holistic, post-hoc step-wise correctness scores. 3. Policy alignment via W-DPO: Preference pairs \((\tau_w, \tau_l, w)\) are constructed by Algorithm 1—solver-distinguished pairs get weight \(w=1.0\); process-only-distinguished pairs get \(w = |r(\tau_{\text{pos}}) - r(\tau_{\text{neg}})|\). The objective is: $\(\mathcal{L}_{\text{W-DPO}}(\theta) = -\mathbb{E}_{(x,\tau_w,\tau_l)}\!\left[w(\tau_w,\tau_l)\cdot\log\sigma\!\left(\beta\log\frac{\pi_\theta(\tau_w|x)}{\pi_\theta(\tau_l|x)}\right)\right]\)$ 4. GenPRM refinement via SFT: Only solver-consistent trajectories (where GenPRM's verdict matches the solver) are used as high-confidence SFT targets to update \(\rho_\phi\).

At inference, GenPRM is used as a Best-of-\(k\) verifier: sequentially sample trajectories and select the first one GenPRM judges process-correct.

Key Contributions

  • First application of generative process supervision to solver-based OR, grounded in numerical feasibility and solver optimality rather than answer-string matching.
  • Co-evolutionary loop between policy and GenPRM—each improvement in one accelerates improvement in the other.
  • Weighted DPO (W-DPO) objective that aggregates fine-grained process scores into a stable trajectory-level signal, avoiding instability of step-wise DPO.
  • GenPRM generalizes as a universal inference-time verifier: substantially boosts ORLM (a separate model) from 65.0% to 75.0% average Pass@1.
  • Release of model weights (policy 8B + GenPRM 8B) and code.

Results

  • StepORLM (8B) achieves 81.4% average Pass@1 across six benchmarks, vs. 75.4% for DeepSeek-V3 (671B), 75.0% for DeepSeek-R1 (671B), 80.3% for OpenAI o3, and 65.0% for ORLM (8B specialized baseline).
  • StepORLM + GenPRM reaches 85.6% average, with +9.9% on ComplexOR and +9.5% on IndustryOR over StepORLM alone.
  • Outperforms all GPT-4o-based agentic methods (OptiMUS-v0.3: 67.1%, CoT: 50.3%, CoE: 65.5%, CAFA: 51.6%).
  • Outperforms larger specialized fine-tuned models: LLMOPT (14B, 65.7%), OptMATH (32B, 81.4% reported from original paper).
  • GenPRM as verifier for ORLM: raises ORLM average from 65.0% to 75.0% (+10pp), vs. +4.9pp from majority voting and +2.0pp from discriminative PRM.
  • Self-evolving iterations show non-monotonic but net-positive gains on IndustryOR, with early iterations fixing structural modeling errors and later iterations fixing code-level bugs.

Limitations

  • The hard subset \(D_{\text{hard}}\) is fixed at 13K problems from the warm-up pool; diversity of training problems is bounded by the teacher LLM's generation capability (GPT-4o).
  • Training requires 8Ă— H100 80GB GPUs; co-evolution adds significant compute versus single-stage SFT.
  • GenPRM refinement uses only solver-consistent trajectories (high-confidence filter), potentially discarding useful signal from ambiguous cases.
  • Evaluation is confined to LP/MILP-style OR benchmarks; generalization to combinatorial or stochastic OR problems is untested.
  • Non-monotonic IndustryOR performance during training suggests sensitivity to dataset characteristics and small test-set variance.
  • Best-of-\(k\) inference scaling still requires multiple policy rollouts per problem, increasing inference cost.

Relevance to Agentic AI / LLM Agents

StepORLM directly challenges the dominant paradigm of using agentic orchestration (e.g., OptiMUS, CoE, CAFA with GPT-4o) for OR problem solving, showing that a deeply trained 8B model with co-evolved process supervision can surpass multi-agent systems built on frontier models. The generative PRM design—reasoning holistically over full trajectories rather than scoring steps in isolation—is a transferable architectural idea for any agentic workflow where intermediate steps have long-range dependencies (planning, code generation, multi-step tool use). The co-evolutionary self-improvement loop, where policy and critic bootstrap each other using external verifier signals, is a concrete instantiation of scalable agent self-improvement that avoids reward hacking documented in related RL-for-agents literature.