Skip to content

LLMSR@XLLM25: Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillation

🕒 Published (v1): 2025-04-23 04:19 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

"Less is More" is a structured multi-agent distillation framework that generates high-quality CoT training data from only 24 labeled examples using reverse-prompt induction, retrieval-augmented synthesis via GPT-4o, and dual-stage reward filtering. Three independent LoRA+-finetuned LLaMA3-8B agents handle question parsing, CoT decomposition, and step-level verification in cascade. The system placed third in the LLMSR@XLLM25 shared task, demonstrating that data quality dominates quantity under low-resource structured reasoning.

Problem

Structured reasoning tasks requiring interpretable, step-level CoT (question parsing → statement decomposition → evidence-grounded verification) are extremely hard to supervise when labeled data is scarce. Existing CoT prompting and instruction-tuning approaches require large corpora and lack structural granularity; they fail when only a handful of seed examples are available and logical consistency across pipeline stages is mandatory.

Method

The pipeline has three stages:

  1. Prompt induction via reverse thinking (RoT): Given seed data \(D^t_\text{seed} = \{(x_i, y_i)\}_{i=1}^N\), an LLM is prompted with a reverse-thinking instruction \(P_\text{reverse}\) to search for an optimal task-specific prompt \(\pi^*_t = \arg\max_{\pi \in \Pi}[S_\text{gen}(\pi) + S_\text{pref}(\pi)]\), balancing generation quality and pairwise preference.

  2. Retrieval-augmented reasoning synthesis: Unlabeled LogiQA instances are annotated at scale by GPT-4o (gpt-4o-2024-08-06). For each question \(x\), the \(k\) nearest seed examples are retrieved via cosine similarity over BGE-M3 embeddings: \(R(x) = \text{TopK}_{x'} \cos(h_x, h_{x'})\). Both a QP prompt \(P_\text{QP}(x)\) and a unified CoT prompt \(P_\text{UCoT}(x)\) are constructed; the output is a structured JSON with cot_steps, evidence, and boolean verification labels.

  3. Dual-stage filtering: (i) Structural pruning removes malformed JSON and traces with fewer than two reasoning steps. (ii) Reward-based selection applies a LLaMA3-based reward model (GRM-Llama3.2-3B) under both few-shot and zero-shot prompting, computing \(s_\text{avg} = \tfrac{1}{2}(s_\text{few} + s_\text{zero})\); only traces with \(S(x) > 0\) are retained. Three filtered subsets (few-shot, zero-shot, average) are produced for ablation.

Three agents—Parser, Decomposer, Verifier—are each independently finetuned from Meta-Llama-3-8B-Instruct using LoRA+ (rank 16, \(\alpha=32\)) and deployed in cascade at inference, each receiving RAG-retrieved demonstrations.

Key Contributions

  • Reverse-prompt induction that bootstraps task-specific instructions from 24 seed examples without manual prompt engineering.
  • RAG-augmented GPT-4o synthesis pipeline that scales structured CoT annotation to thousands of examples from a minimal seed pool.
  • Dual-prompt reward filtering (few-shot + zero-shot averaging) that selects semantically faithful traces, demonstrably outperforming structural filtering alone.
  • Evidence that reward signals on intermediate CoT modules transfer emergent quality gains to uninvolved upstream modules (Question F1 rises even though QP is not in the reward loop).
  • Modular multi-agent inference cascade (Parser → Decomposer → Verifier) with consistent RAG grounding across all agents.

Results

All comparisons are on the LLMSR@XLLM25 public test set, with training data fixed to same-size subsets differing only in filtering strategy:

  • Structure filtered only: Ques. F1 56.87 / Stmt. F1 36.72 / Evid. F1 10.80 / Reason. F1 5.20
  • 0-shot reward filter: Ques. F1 62.76 / Stmt. F1 38.05 / Evid. F1 12.79 / Reason. F1 7.15
  • 5-shot reward filter: Ques. F1 65.89 / Stmt. F1 38.26 / Evid. F1 14.45 / Reason. F1 7.70
  • Avg. reward filter (best): Ques. F1 66.71 / Stmt. F1 39.21 / Evid. F1 14.92 / Reason. F1 8.98
  • Avg. reward improves Reasoning F1 by +3.78 pp, Evid. F1 by +4.12 pp, and Ques. F1 by +9.84 pp over structure-only filtering.
  • System ranked 3rd in LLMSR@XLLM25 shared task.

Limitations

  • Reasoning F1 remains low in absolute terms (8.98), indicating the task is far from solved even with quality distillation.
  • GPT-4o synthesis introduces cost and API dependency; not fully self-contained.
  • All fine-tuned agents are fixed at LLaMA3-8B; scaling behavior with larger backbone models is untested.
  • Reward model (GRM-Llama3.2-3B) is itself an approximate scorer; its calibration under domain shift is not validated.
  • Evaluated on a single shared-task dataset (LogiQA-derived); generalizability to other structured reasoning domains is not demonstrated.
  • The cascade agent architecture has no error-recovery mechanism; parse failures in early agents propagate unchecked.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a distillation meta-harness: a multi-stage pipeline that orchestrates prompt induction, synthetic data generation (via a strong teacher LLM), filtering, and modular fine-tuning to produce specialized agents from minimal seed supervision. The three-stage harness (induce → synthesize → filter) mirrors the structure of meta-harnesses that manage agent lifecycle and quality control automatically. The dual-prompt reward filtering sub-system acts as an inline quality gate—a pattern common to harness designs that route or prune agent outputs before downstream consumption. For researchers tracking harnesses, this work demonstrates how a harness can bootstrap its own training data pipeline, reducing dependence on human annotation at scale while maintaining structural coherence across pipeline stages.