Skip to content

rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection

🕒 Published (v1): 2025-12-09 06:55 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

rSIM introduces a small "planner" LLM trained via multi-agent RL to adaptively inject reasoning strategies (self-reflection, decomposition, etc.) into another LLM's chain-of-thought at each reasoning step. This leader-follower MARL setup lets even 0.5B models achieve accuracy comparable to 14B models, bypassing the fundamental failure of pure RL post-training on weak base models. The trained planner is pluggable—deployable on unseen LLMs without additional training.

Problem

Pure RL post-training (e.g., GRPO) induces "aha moments"—emergent reasoning strategies—only in models that already exhibit some baseline strategy usage. Models ≤0.5B parameters show zero inherent strategy frequency; training reward collapses to 0 rather than inducing reasoning, leaving small LLMs unable to benefit from RL-based reasoning enhancement.

Method

A planner LLM (leader agent) is attached to a reasoner LLM (follower agent). At each reasoning step, the planner selects one of nine human-defined strategies (self-reflection, decomposition, deliberative thinking, validation, summarization, prioritization, continuation, sub-planning, termination) and injects it as a prompt prefix guiding the next step. The planner's action head is a linear layer (hidden-dim × 9) appended to a decoder-only LLM; the highest-score strategy is selected greedily during inference.

Training uses a leader-follower MARL objective (Stackelberg equilibrium via GRPO-style updates) with a two-stage scheme: stage 1 sets λ=0.7 (emphasize planner) then stage 2 λ=0.3 (emphasize reasoner) to avoid conflicting gradients and credit assignment ambiguity. Planner rewards include accuracy (Racc), terminal strategy signal (Rterminal), and a diversity penalty (−ratio of most-frequent strategy). Reasoner rewards include accuracy, format, and a strategy-following ratio (Rfollow).

Key Contributions

  • Demonstrates that pure RL fails on sub-1B LLMs due to absent baseline strategy usage, and quantifies the positive correlation between strategy frequency and accuracy.
  • Proposes rSIM: adaptive step-wise strategy injection via a lightweight planner LLM trained jointly with a reasoner under MARL leader-follower framework.
  • Shows the trained planner is pluggable: a planner trained once on MATH transfers to different LLM families (Qwen2.5 → Llama) and tasks without additional training.
  • Demonstrates continual learning: planners can be sequentially fine-tuned on new task domains while retaining prior performance.

Results

  • Qwen2.5-0.5B+rSIM (0.5B planner) matches Qwen2.5-14B accuracy on MATH; Qwen2.5-0.5B alone scores ~10–20% lower.
  • All Qwen2.5+rSIM7B variants achieve SOTA on MATH, surpassing GRPO-trained counterparts at every model size (0.5B, 1.5B, 7B).
  • Plugin mode: Qwen2.5-7B+Planner outperforms Llama-3.1-405B-Inst on AIME2024.
  • Llama3.2-1B+rSIM (1B planner): TheoremQA 61.5 vs. 30.6 baseline; approaches Llama3.3-70B (78.3).
  • Continual training on CodeAlpaca-20k yields +17–24.4% HumanEval gain across Qwen2.5-0.5B/7B reasoners with only −0.8% regression on MATH.
  • Evaluated across 7 datasets: MATH, GSM8K, AIME2024, MMLU-Pro, TheoremQA, CodeAlpaca-20k, HumanEval.

Limitations

  • Nine reasoning strategies are hand-designed; optimality and completeness of this set are not validated.
  • Continual training on coding yields zero gain on AIME2024, suggesting domain-specific limits to transfer.
  • Cross-model improvement magnitude varies with base model quality; weakest models (0.5B as planner for 70B reasoner) show smaller relative gains.
  • Strategy counting relies on keyword matching, which is a noisy proxy for actual strategy execution quality.
  • No ablation on planner architecture size vs. performance trade-off across all tasks.

Relevance to Agentic AI / LLM Agents

rSIM is a concrete instantiation of a two-agent MARL system where one agent (planner/leader) controls the meta-policy of another (reasoner/follower)—directly relevant to hierarchical and modular agent architectures. The pluggable planner paradigm proposes a new decomposition: separate the "planning intelligence" module from the "execution LLM," enabling capability injection without fine-tuning the base model, which is significant for deploying reasoning-capable agents on compute-constrained systems. The continual learning result shows planners can accumulate task-specific planning knowledge across domains, relevant to lifelong agentic systems. This builds on the MARL multi-agent LLM literature (ReAct, CORY, MetaGPT) but is distinctive in using RL-trained strategy selection rather than prompted coordination.