Skip to content

AgentODRL: A Large Language Model-based Multi-agent System for ODRL Generation

🕒 Published (v1): 2025-11-29 19:19 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentODRL is a multi-agent LLM system using an Orchestrator-Workers architecture to convert natural language data-rights policies into W3C ODRL format. It combines a LoRA-finetuned semantic reflection model with a SHACL-based iterative syntax validator to achieve near-perfect grammatical compliance and substantial semantic fidelity gains over prior monolithic approaches.

Problem

Existing NL-to-ODRL methods use a single LLM end-to-end, which cannot concurrently handle the cognitively distinct sub-tasks of parsing recursive/parallel legal structures, semantic segmentation, and constrained code generation. Compounding this, no public benchmark dataset existed for ODRL generation, and real-world policies with complex cross-clause dependencies (e.g., GDPR article references) cause severe accuracy degradation in prior methods.

Method

AgentODRL employs an Orchestrator-Workers pattern with three specialized worker agents: - Rewriter Agent: resolves cross-clause (recursive) dependencies via "structure-preserving inlining" — explicitly and implicitly referenced clauses are inlined to eliminate semantic dependency before further processing. - Splitter Agent: decomposes parallel-structure policies into independent semantic units by detecting shifts in core asset, assigner-assignee relationship, or policy purpose; assigns ODRL type (Agreement/Offer/Set) via heuristics. - Generator Agent: produces ODRL output with two quality layers: (1) a LoRA-finetuned Qwen3-4B model (r=16, alpha=32, trained on 2,380 synthetic samples) acts as a semantic checkpoint extractor — the generator must validate its output against extracted key elements; (2) a PYSHACL-based SHACL validator drives a generate-validate-correct loop until syntax passes or max iterations (8) are reached.

The Orchestrator dynamically classifies input complexity and routes through the appropriate path: Simple → Generator only; Parallel → Splitter → Generator; Recursive → Rewriter → Splitter → Generator.

Key Contributions

  • AgentODRL framework: first MAS applied to ODRL generation, combining Orchestrator-Workers architecture with syntactic and semantic enhancement modules.
  • LoRA-finetuned semantic reflection: a lightweight expert validator (Qwen3-4B) extracted as a "semantic checkpoint" oracle to enforce semantic fidelity during generation.
  • SHACL-based iterative syntax correction: closed-loop validator ensuring W3C ODRL compliance, achieving near-perfect grammar scores (>99%).
  • New benchmark dataset: 770 NL-to-ODRL use cases spanning simple (440), parallel (220), and recursive (110) structural complexities, constructed from seed cases drawn from GDPR, CCPA, Creative Commons 4.0, and augmented via Gemini 2.5 Pro.

Results

Compared to prior SOTA (SCR-Enhanced), averaged across GPT-4.1, GPT-4.1-mini, GPT-4.1-nano on 770 cases: - +5.39% grammar score (avg across all models); up to +7.32% for GPT-4.1-nano on recursive cases. - +14.52% semantic score (avg); up to +28.67% for GPT-4.1-nano overall. - Grammar scores for AOFP (AgentODRL Full Pipeline) consistently exceed 99% across all models and complexity levels. - GPT-4.1-nano semantic score on recursive cases: 34.87 (OGS baseline) → 61.53 (AOFP), a +76.46% relative improvement. - Orchestrator-Workers workflow (Experiment 2, GPT-4.1-nano generator): semantic score 80.22 vs. 72.35 for Generator-only baseline; token cost 46.2M vs. 47.9M (Splitter path) and 49.5M (full Rewriter path), demonstrating efficiency-performance balance. - Semantic scores degrade sharply without specialized agents: parallel cases 69.44 (Generator only) → 84.88 (Splitter+Generator); recursive cases 61.53 → 82.00 (Rewriter+Splitter+Generator).

Limitations

  • Dataset of 770 cases is synthetically augmented from only 70 seed cases using Gemini 2.5 Pro; distribution may not reflect full real-world policy diversity.
  • Orchestrator-Workers automated routing slightly underperforms manually-selected optimal fixed paths (e.g., 80.22 vs. 84.02–88.07 for Experiment 2's best fixed paths), indicating routing errors exist.
  • Evaluation relies on an LLM jury (K=2) for semantic scoring, which may introduce model-specific biases.
  • Maximum reflection count (8) is a fixed hyperparameter; weaker models (GPT-4.1-nano) average 6–7 reflections on recursive cases, approaching the cap.
  • No evaluation on non-data-space ODRL use cases (e.g., media licensing, IoT); generalizability to other ODRL application domains is untested.

Relevance to Agentic AI / LLM Agents

AgentODRL exemplifies the Orchestrator-Workers multi-agent pattern applied to a structured-output generation task requiring strict formal compliance, directly relevant to the line of work on specializing agent roles for cognitively distinct sub-tasks (cf. MetaGPT, ChatDev). The paper's core argument — that task decomposition by structural complexity yields better results than monolithic LLM prompting — provides concrete evidence for when and why agent specialization outperforms single-model approaches. The pairing of a LoRA-finetuned lightweight validator as a domain-expert sub-agent alongside a larger generative LLM is a practical instantiation of the generator-critic pattern. The SHACL-based iterative correction loop demonstrates how tool-augmented agents (external validators providing error feedback) can enforce hard correctness constraints that pure LLM generation cannot guarantee.