Skip to content

QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management

🕒 Published (v1): 2025-12-15 04:11 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

QwenLong-L1.5 is a complete post-training recipe for long-context reasoning built on Qwen3-30B-A3B-Thinking, combining a scalable data synthesis pipeline, stabilized RL training, and a memory-agent framework for ultra-long (1M–4M token) inputs. It achieves a 9.90-point average gain over its baseline across six long-context benchmarks, reaching parity with GPT-5 and Gemini-2.5-Pro. A separate memory-agent inference harness adds a further 9.48-point gain on tasks exceeding the 256K context window.

Problem

Post-training for long-context reasoning lacks end-to-end solutions: existing work focuses on pre-/mid-training or architecture changes, leaving no mature recipe covering (1) scalable synthesis of genuinely hard multi-hop long-context data, (2) RL training methods robust to the instability specific to long-context multi-task settings, and (3) an agent architecture for inputs exceeding the model's physical context window.

Method

Data Synthesis Pipeline: Documents are deconstructed into atomic facts; knowledge graphs guide multi-hop QA generation with entity obfuscation and cross-document path sampling. A Structural Tabular Data Engine generates corpus-level numerical reasoning via NL2SQL execution. A Multi-Agent Self-Evolved (MASE) framework (proposer → solver → verifier loop with a history buffer) produces general long-context tasks. Two quality gates filter outputs: Knowledge Grounding Check (question must require the document) and Contextual Robustness Check (answer must survive irrelevant-document injection). Final set: 14.1K samples from 42.7K synthesized.

RL Training: Multi-stage length extension (32K→60K→120K input, with proportionally scaled max output). Two multi-task RL fixes: (a) task-balanced sampling stratified by pass@k bins, and (b) task-specific advantage estimation replacing group- or batch-level normalization with per-task-type reward standard deviations. AEPO (Adaptive Entropy-Controlled Policy Optimization) dynamically gates negative-advantage samples based on batch entropy: when entropy > H_high, only positive samples update the model (rejection sampling mode); when < H_low, negative gradients re-enter to prevent collapse.

Memory Architecture: The context is chunked; at each step the policy updates a compact memory m_t and a navigational plan p_t before processing chunk x_{t+1}. Final answer is generated from accumulated memory m_K plus formatting instructions. Memory-RL is trained separately as an expert, then merged with the full-context model via SCE (Spherical Consistent Extrapolation), followed by a fourth full-context RL stage to stabilize both capabilities.

Key Contributions

  • End-to-end post-training recipe unifying data synthesis, RL training, and inference-time agent architecture for long-context reasoning.
  • Knowledge-graph-guided multi-hop QA synthesis with entity obfuscation and cross-document subgraph sampling.
  • Task-balanced sampling + task-specific advantage estimation to stabilize multi-task long-context RL (+2.55 pts over GRPO baseline on 4B model).
  • AEPO: entropy-adaptive masking of negative-advantage sequences during online RL (+3.29 pts over GRPO baseline on 4B model).
  • Multi-stage expert training + SCE model merging to integrate memory-agent and full-context capabilities without mutual degradation.
  • Memory agent trained with trajectory-level GRPO, enabling processing of 1M–4M token inputs within a 256K-window model.

Results

  • QwenLong-L1.5-30B-A3B vs. Qwen3-30B-A3B-Thinking-2507 baseline: +9.90 pts average across MRCR, CorpusQA, DocMath, LBV1-QA, Frames, LBV2 (83.0 vs. 71.2 average score in Figure 1).
  • Surpasses DeepSeek-R1-0528, Gemini-2.5-Flash-Thinking, Qwen3-Max-Thinking; comparable to Gemini-2.5-Pro.
  • Memory-agent framework on 1M–4M token tasks: +15.26 pts over baseline single-pass, +9.48 pts over baseline memory-agent configuration.
  • AEPO ablation (Qwen3-4B-Thinking): 59.36 avg vs. 56.07 for GRPO (+3.29 pts).
  • Task-balanced sampling + task-specific advantage estimation (4B): 58.62 avg vs. 56.07 (+2.55 pts), with pronounced gains on MRCR (dense-reward task).
  • Benefits generalize to mathematics, natural science QA, tool-using agents, and long dialogue.

Limitations

  • Memory agent architecture is adopted from prior work (Yu et al. 2025a), not novel in design.
  • Training data synthesis quality depends on LLM judge (gpt-oss-120b) for reward, introducing dependency on a proprietary external model.
  • SCE merging and four-stage training pipeline are computationally expensive; ablations are run on the smaller 4B model to proxy the 30B model's behavior.
  • Upper limit of memory-agent evaluation is 4M tokens; behavior at even longer sequences is not characterized.
  • The text is truncated before the full evaluation section, so some benchmark details and baselines cannot be verified from the provided excerpt.

Relevance to Harnesses / Meta-Harnesses

The memory-agent framework is a canonical inference-time meta-harness: it wraps a base LLM in a sequential chunk-processing loop with explicit state (memory) and planning (navigational plan), trained end-to-end via trajectory-level RL — a direct instance of a learnable agentic harness rather than a hand-coded pipeline. The multi-stage training pipeline itself is a post-training harness that composes data synthesis agents (MASE's proposer/solver/verifier), progressive RL stages, and model merging into a single recipe. AEPO is an adaptive control mechanism inside that harness that modulates the RL objective based on observed training dynamics, demonstrating how meta-level monitoring can stabilize inner-loop learning. This work is directly relevant to researchers studying how to train models that are aware of and robust within harness-structured inference regimes.