Skip to content

Domain-Partitioned Hybrid RAG for Legal Reasoning: Toward Modular and Explainable Legal AI for India

🕒 Published (v1): 2025-12-23 20:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a domain-partitioned hybrid RAG–Knowledge Graph system for Indian legal AI, combining three specialized retrieval modules (case law, statutes/constitution, IPC) with a Neo4j KG and an LLM-driven agentic orchestrator. The hybrid system achieves a 70% pass rate on a 40-question LLM-as-a-Judge benchmark versus 37.5% for a RAG-only baseline. The architecture targets India's specific legal heterogeneity—cross-referenced statutes, penal codes, and hierarchical precedents—that monolithic retrievers fail to handle.

Problem

Existing legal AI platforms (SCC Online, Manupatra, LexisNexis) rely on keyword retrieval and produce weak contextual grounding. Monolithic or purely embedding-based RAG pipelines suffer from retrieval drift, citation chaining failures, and inability to perform multi-hop reasoning across structurally heterogeneous Indian legal sources (statutes, IPC, case law). No prior system unified semantic retrieval with structured relational reasoning under an agentic controller for the Indian jurisdiction.

Method

The system partitions the legal corpus into three independent ChromaDB vector stores, each with a dedicated RAG module: - Past Cases RAG: ~2,900 Supreme Court judgments chunked by section headers (Facts/Issues/Held) - Statutes & Constitution RAG: PDF-parsed constitutional articles + Central Acts CSVs - IPC RAG: 3,481 FIR/offense records normalized into structured text templates

All embeddings use all-MiniLM-L6-v2 (384-dim) with HNSW indexing and cosine similarity. A Neo4j Knowledge Graph (2,586 nodes, 5,056 relationships) captures entity types (Case, Judge, Article, IPCSection, Offense, Punishment) and relations (HEARD_IN, DECIDED_BY, GOVERNED_BY, REFERS_TO, APPLIES_TO), enabling multi-hop Cypher traversals.

An LLM-driven agentic orchestrator (built on LangChain + asyncio) classifies queries by domain, routes to one or more RAG modules and/or KG traversal in parallel, then fuses evidence into grounded answers via Gemini 2.5 Flash (temperature=0.1, 8192-token context). Prompts enforce explicit citation constraints to reduce hallucination.

Evaluation uses a 40-question synthetic benchmark (5 legal domains) scored by an LLM-as-a-Judge across Correctness, Completeness, Relevance, and Legal Reasoning Quality (each 0–10), with PASS threshold yielding a binary pass rate.

Key Contributions

  • Domain-partitioned retrieval strategy with three independent RAG modules, each optimized for structurally distinct Indian legal document types
  • Neo4j KG with 2,586 nodes and 5,056 relationships linking statutes, IPC sections, cases, and judges, enabling relational multi-hop queries unsolvable by embedding similarity
  • LLM-based agentic orchestrator for dynamic query classification, parallel multi-module retrieval, and grounded evidence fusion
  • 40-question synthetic LLM-as-a-Judge benchmark grounded in the system's own ingested corpus, spanning constitutional, criminal, statutory, citation, and cross-domain reasoning

Results

  • Hybrid (RAG+KG) vs. RAG-only: 70% pass rate (6.09/10 avg) vs. 37.5% (4.16/10 avg)
  • Completeness: 4.97 vs. 2.67 (largest margin); Legal Reasoning: 5.35 vs. 3.95; Correctness: 6.10 vs. 4.03; Relevance: 8.15 vs. 6.00
  • IPC RAG strongest individual module: F1@5=0.402, ROUGE-L=0.467, Token Overlap=0.402
  • Constitution/Acts RAG: high precision@5 (0.880) but low recall@5 (0.140), F1@5=0.224
  • Past Cases RAG: moderate precision@5 (0.580), low recall@5 (0.150), F1@5=0.175
  • KG retriever: Precision=0.494, Recall=0.300, F1=0.222, ROUGE-L=0.653, Token Overlap=0.653 (highest grounding scores)

Limitations

  • KG is small (2,586 nodes, 5,056 relations) covering limited Supreme Court data; no High Court coverage
  • RAG corpora are small curated sets (~2,900 cases, 3,481 IPC records), not comprehensive statutory collections
  • 40-question evaluation benchmark is synthetic and self-referential (grounded in the system's own ingested data), limiting generalizability
  • Orchestrator struggles with open-ended legal reasoning and legal analytics beyond structured KG queries
  • No automated ingestion pipeline; entity extraction via LLM calls is non-production quality
  • No provenance tracking, confidence calibration, or fallback mechanisms in current prototype
  • Evaluation uses a single judge model with no inter-rater reliability reported

Relevance to Harnesses / Meta-Harnesses

This paper demonstrates a concrete multi-module orchestration harness pattern: an LLM-driven controller that dynamically routes queries across heterogeneous retrieval backends (three domain-specialized RAGs plus a graph engine) and fuses results—precisely the kind of agentic meta-harness architecture this research track monitors. The LLM-as-a-Judge evaluation framework also illustrates how harnesses can incorporate automated end-to-end quality assessment pipelines that score multi-component system outputs without human annotation at scale. The modular, stateless pipeline design (each RAG exposes a uniform API) is directly applicable to building composable harness components. The 70% vs. 37.5% pass rate delta quantifies the benefit of adding a structured reasoning layer on top of a vanilla retrieval harness.