Skip to content

Enhancing Small Language Models Reasoning through Knowledge Graph Grounding

🕒 Published (v1): 2026-07-14 13:07 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper wraps Small Language Models (Gemma 3 1B/4B, Llama 3.2 3B) in a two-tool agentic loop—extract_facts for symbolic triplet extraction and get_hint via a pre-trained RGCN—to improve multi-hop kinship reasoning on the CLUTRR benchmark. RGCN-derived hints yield a 1.5–2× accuracy gain over story-only baselines but the pipeline is bottlenecked by noisy SLM extraction, which collapses RGCN accuracy from 60.69% to 24.88%. The work characterizes the "extraction bottleneck" and "distraction effect" as the core failure modes of neuro-symbolic SLM pipelines.

Problem

SLMs (1B–4B parameters) fail at multi-hop relational reasoning because they cannot maintain a consistent symbolic state over long contexts; errors in early reasoning steps compound, causing catastrophic failure at higher hop counts. Existing fixes—multi-agent architectures, LoRA fine-tuning, knowledge distillation—re-introduce cost or require supervised data, defeating the low-resource motivation.

Method

The SLM acts as a single-agent reasoning controller that invokes two tools in sequence:

  1. extract_facts(story) — zero-shot structured prompting forces the SLM to emit kinship triplets \((u, \text{rel}, v)\) in a target-centric reversed-edge schema (object, rel, subject). Inverse triplets are appended post-hoc to ensure bidirectional graph connectivity.
  2. get_hint(K\_graph, target\_pair) — the extracted graph is fed to a 4-layer RGCN trained on CLUTRR training stories (\(k \in \{2,3,4\}\) hops). The RGCN returns the top predicted relation and a confidence score \(\sigma\), injected into the SLM's context before final answer generation.

Two evaluation regimes are used: Oracle (ground-truth triplets provided) and Realistic (SLM self-extracts). Models tested zero-shot; no fine-tuning of SLMs.

Key Contributions

  • Minimalist two-tool agentic harness for SLMs that decouples text understanding (SLM) from relational inference (RGCN).
  • Comparative study across three open-weight architectures (Gemma 3 1B, Gemma 3 4B, Llama 3.2 3B), isolating the role of model size on extraction quality.
  • Empirical characterization of the extraction bottleneck: RGCN accuracy drops from 60.69% (Oracle facts) to 24.88% (SLM-extracted facts, ~71% ground-truth recall).
  • Identification of the distraction effect: for Gemma 4B, Story + GNN Hint (19.75%) outperforms Story + SLM Facts + GNN Hint (14.98%), showing noisy facts override correct expert signals.
  • Evidence that Llama 3.2 3B is more robust to symbolic noise than Gemma variants, suggesting architecture-specific differences in attention-based filtering.

Results

  • Gemma 1B story-only baseline: 7.54% accuracy; with GNN hint (realistic): 12.60% (~1.67Ă—).
  • Llama 3B story-only: 16.13%; with Oracle Hint: 62.79% (~4Ă— gain); with GNN hint (realistic): 20.32%.
  • Gemma 4B story-only: 13.45%; with Oracle Hint: 59.16%; with GNN hint only (realistic): 19.75%; with SLM facts + GNN hint: 14.98% (distraction effect — hints hurt when noisy facts are co-present).
  • RGCN expert: 60.69% accuracy on Oracle facts; 24.88% on SLM-extracted facts.
  • Realistic configurations show a "Reasoning Cliff" beyond 4 hops, with most models below 5% accuracy even for 2-hop chains when extraction fails.
  • Llama-based models maintain or improve accuracy beyond 4 hops in Oracle configurations; Gemma variants decay sharply.

Limitations

  • ~71% average recall of ground-truth facts under realistic SLM extraction; no closed-loop verification or correction of extraction errors.
  • RGCN trained only on \(k \in \{2,3,4\}\) hops; systematic generalization to 10 hops is partially evidenced but not guaranteed.
  • Evaluated exclusively on CLUTRR (synthetic kinship); generalizability to open-domain multi-hop tasks is undemonstrated.
  • The agentic loop is strictly sequential with no iterative refinement or retry on extraction failure.
  • No fine-tuning of SLMs to improve extraction fidelity; the tool-calling prompt must fit constrained context windows.
  • Only three model architectures tested; conclusions about "Llama vs. Gemma" resilience are suggestive, not definitive.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct case study in single-agent tool-calling harness design: the two-tool loop (extract_facts → get_hint) is a minimal harness that wraps an SLM with external symbolic modules, and the paper's central empirical finding—that harness performance is ceiling-capped by the extraction step, not the reasoning module—is a critical design insight for any multi-tool orchestration pipeline. The "distraction effect," where injecting noisy intermediate outputs degrades final performance despite a correct downstream signal, directly informs harness design choices around when to gate or filter intermediate tool outputs before re-injection into the LLM context. The Oracle-vs-Realistic ablation methodology is a reusable evaluation pattern for harness auditing: it cleanly separates tool quality from reasoning quality, which is exactly the diagnostic a meta-harness designer needs.