Learning to Refine: An Agentic RL Approach for Iterative SPARQL Query Construction

🕒 Published (v1): 2025-11-14 08:44 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A 3B-parameter LLM trained via GRPO reinforcement learning (no SFT) learns to iteratively construct and debug SPARQL queries against a live knowledge graph through a think–act–observe loop. On a curated LC-QuAD 2.0 subset, it achieves 49.7% accuracy—a 17.5 pp gain over the best zero-shot iterative baseline. The key insight is that outcome-driven RL alone suffices to teach symbolic tool mastery, with an explicit <think> scaffold further sharpening policy precision.

Problem

One-shot SPARQL generation by LLMs is brittle for multi-hop KGQA: syntax errors, empty results, and wrong relation paths cascade with no recovery mechanism. Existing agentic KGQA frameworks (ReAct, StructGPT) use static prompting over base models and do not learn adaptive interaction policies from execution feedback.

Method

The agent operates as a finite-horizon MDP. At each turn, the policy \(\pi_\theta\) (Qwen2.5-3B-Instruct + QLoRA adapters) generates a <think> block followed by either <query> (SPARQL) or <answer>. A self-hosted qEndpoint (Wikidata HDT dump) executes queries and returns <query_result> tokens appended to the conversation state. Episodes run up to \(T_{\max} = 10\) turns.

Training uses GRPO with \(G = 16\) rollouts per question. The terminal composite reward is:

\[R(\tau) = \begin{cases} -1 & \text{if structurally invalid} \\ 1 + R_{\text{ans}}(\tau) - (0.1\,N_{\text{err}} + 0.02\,T) & \text{otherwise} \end{cases}\]

where \(R_{\text{ans}} \in \{+0.5, -0.2\}\) based on a GPT-4o-nano judge, and the cost term penalizes SPARQL failures (\(N_{\text{err}}\)) and turn count (\(T\)). Loss masking excludes environment tokens from gradient updates, focusing learning on the agent's own output tokens only. No SFT warm-up phase is used.

Key Contributions

• First demonstration that a 3B-parameter LLM can learn multi-turn SPARQL refinement from sparse outcome rewards alone (no gold trajectories).
• MDP formalization of iterative KGQA with a composite reward that enforces structural validity, answer correctness, and interaction efficiency.
• Ablation isolating the role of <think> reasoning: Reactive agent (no <think>) reaches 48.1% vs. Deliberative agent's 49.7%, confirming RL is the primary driver but explicit reasoning adds a measurable scaffold effect.
• Error analysis documenting a qualitative failure-mode shift: from syntactic failures (57% of zero-shot failures) to semantic/logic errors (72.5% of RL agent failures).
• Private containerized SPARQL environment with LRU caching, async execution, and pre-flight syntax checks, sustaining training-scale query throughput.

Results

• RL-Tuned Agent: 49.7% accuracy vs. 32.2% (B3: prompt-guided iterative agent), 19.7% (B2: one-shot SPARQL), 16.3% (B1: direct CoT) on curated LC-QuAD 2.0 test set (N=1,279).
• Query executability: 81.0% (RL) vs. 54.7% (B3) vs. 47.7% (B2).
• Pass@5: 77.7% (RL) vs. 61.7% (B3) vs. 47.2% (B2).
• Statistical significance: McNemar's test \(\chi^2(1) = 102.75\), \(p \ll .001\) over 354 RL-correct/baseline-wrong vs. 130 RL-wrong/baseline-correct discordant pairs.
• Training: 11.5 hours on 1× H100 (94 GB), ~4.6 kWh, ~1.7 kg CO₂e.

Limitations

• Evaluated only on single-answer questions (curated subset: 5,112 train / 1,279 test from original 24,180); multi-answer aggregation is out of scope.
• Relies on pre-linked entities (gold entity linking provided); end-to-end entity disambiguation not addressed.
• No systematic hyperparameter tuning; rank, \(\alpha\), and learning rate chosen by manual trial.
• Tested on a single KG (Wikidata); generalization to other SPARQL endpoints or graph schemas is unverified.
• Scalability and energy efficiency at larger model sizes or across domains not evaluated.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a tool-use harness: a structured loop (think → act → observe) with environment execution, reward shaping, and policy learning layered over a base LLM. The loss-masking design—training only on agent-generated tokens while treating environment responses as opaque observations—mirrors a key principle in meta-harness design: cleanly separating the learnable controller from the deterministic execution substrate. The reward function itself functions as a meta-level policy shaping layer, decomposing task success into structural, semantic, and efficiency sub-signals that guide multi-step agent behavior without step-level supervision. For researchers building harnesses, the demonstration that GRPO over sparse terminal rewards can teach a compact model complex multi-turn tool use without SFT demonstrations has direct implications for reducing annotation overhead in harness training pipelines.