Skip to content

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

🕒 Published (v1): 2026-07-20 14:59 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NLKGQ is a reusable framework that enables zero-shot natural language querying of domain-specific metadata archives by coupling a well-designed OWL ontology with a domain-agnostic LLM harness that translates NL questions to SPARQL. Evaluated on a neuroimaging archive (80 studies, 2,000 experiments, ~10M RDF triples), the best model (Qwen3.6-27B) achieves 100% accuracy on SPARQL versus 57% on auto-generated SQL. The dominant accuracy factor is ontology design quality—specifically readable entity names and semantic annotations—rather than model choice or prompt engineering.

Problem

Researchers querying domain-specific metadata archives must write SPARQL or SQL, requiring both query-language expertise and domain vocabulary knowledge. Prior LLM-driven query generation approaches either target public knowledge graphs with opaque identifiers (requiring fine-tuning, RAG, or multi-agent orchestration) or treat the schema as fixed and optimize around it, rather than co-designing the ontology to maximize LLM comprehension. No prior work provides an end-to-end reusable process: ontology design → ETL → NL querying, evaluated with a controlled SPARQL-vs-SQL backend comparison over identical data and questions.

Method

The ontology-first development process consists of three iterative steps: 1. OWL ontology design following six LLM-legibility principles: full English words, consistent has<Property> naming, descriptive relationship names, explicit rdfs:domain/rdfs:range, natural language annotations (rdfs:comment, rdfs:label, skos:altLabel), and no opaque codes. 2. Domain-specific ETL pipeline that extracts metadata from archive sources (DICOM, NIfTI, experiment registries) and materializes RDF triples (including inverse properties and class memberships) into Apache Jena Fuseki for SPARQL 1.1 queries. 3. Iterative co-evolution of the ontology, competency questions, and a domain-specific prompt rider until test accuracy converges.

The NLKGQ query server (the domain-agnostic harness) assembles a system prompt from: (a) generic SPARQL generation instructions, (b) the full OWL Turtle ontology (~80KB, ~18K tokens), and (c) the domain-specific prompt rider. Post-generation, the harness strips chain-of-thought <think> blocks, extracts SPARQL from markdown, corrects PREFIX declarations against the ontology, and optionally retries on syntax errors (up to two attempts).

For the SQL comparison, a generic OWL-to-SQL converter automatically derives a PostgreSQL schema (OWL class → table, datatype property → column, object property → foreign key), preserving ontology annotations as SQL column comments. The same competency questions are evaluated against both backends.

Evaluation uses a combinatorial test driver framework: 8 models × 8 ontology representations × 3 prompts × 4 temperatures = 16,000+ SPARQL runs; 8 models × 2 SQL schema variants × 4 temperatures for SQL. Eight Qwen3 variants (8B–35B, dense and MoE, including quantized variants) are served locally via vLLM and llama.cpp on institutional hardware (AMD MI300A HPC and 4× RTX 5000) to satisfy GDPR constraints on human subject data.

Key Contributions

  • NLKGQ framework: domain-agnostic query server, web interface, and combinatorial test driver, demonstrated on an MRI neuroimaging archive
  • Six OWL design principles for LLM-legible ontologies, validated as the primary driver of accuracy
  • Ablation study across eight ontology representations showing semantic annotations outweigh model selection or prompt engineering
  • Generic OWL-to-SQL schema generator enabling controlled SPARQL-vs-SQL comparison over identical data and NL questions
  • Privacy-compliant local deployment: identification of the smallest model achieving acceptable accuracy on modest institutional GPU hardware

Results

  • Best configuration (Qwen3.6-27B dense, full Turtle ontology): 100% accuracy on SPARQL, 57% on auto-generated SQL
  • Stripping ontology annotations (no-comments representation) degrades SPARQL accuracy by 19 percentage points and SQL by 14 pp, the largest single ablation drop
  • Scale within architecture: 8B → 29%, 14B → 38%, 27B dense → 100% (SPARQL)
  • MoE architectures (35B/3B active) peak at 57–62% on SPARQL, well below 27B dense despite larger parameter count, suggesting MoE is less effective for strict schema-adherence tasks
  • Q8-quantized 27B on vintage RTX 5000 hardware matches full-precision accuracy at 1.8Ă— slower throughput
  • compact-grouped and compact-typed representations retain substantial accuracy; abstract-graph (fully anonymized) collapses to near-zero
  • SPARQL consistently outperforms SQL across all models and configurations, replicating Sequeda et al.'s 16%→54% finding in a controlled same-data setting

Limitations

  • Evaluation domain is a single neuroimaging archive; generalizability to other domain-specific ontologies is claimed but not empirically demonstrated
  • Competency question set is small (>20 questions developed with domain experts); coverage of edge cases and adversarial phrasings is not characterized
  • Retry logic (up to 2 LLM correction passes on syntax errors) breaks strict zero-shot framing for smaller models
  • MoE architecture underperformance is observed but not mechanistically explained
  • Paper text truncated before all competency questions are listed (Table 2 cut off), preventing full replication assessment
  • Approach requires ontology authorship expertise or LLM-assisted ontology creation; cost of initial domain modeling is not measured
  • No evaluation on unseen ontologies or transfer to a second domain

Relevance to Harnesses / Meta-Harnesses

The NLKGQ query server is itself a domain-agnostic harness: it wraps LLM calls with prompt assembly, response post-processing (CoT stripping, markdown extraction, PREFIX repair), and retry logic, insulating end-users and domain-specific artifacts from the raw LLM interface. The test driver is a meta-harness that orchestrates combinatorial sweeps across models, ontology representations, prompts, temperatures, and backends—over 16,000 runs—logging structured results for ablation analysis, a pattern directly analogous to evaluation meta-harnesses in agent benchmarking. A key insight for harness designers is that context quality fed to the harness (ontology readability) dominates accuracy more than harness-level prompt engineering, shifting the design locus upstream. The framework's explicit separation of domain-specific artifacts (ontology, ETL, prompt rider) from the domain-agnostic harness layer is a clean instantiation of the pluggable harness pattern increasingly prominent in agentic system design.