CHARLIE: An On-Premise Multi-Agent Retrieval-Augmented Generation System for Evidential Reasoning in Forensic Science

🕒 Published (v1): 2026-07-01 14:14 UTC · Source: Arxiv · link

Why this paper was selected

On-premise multi-agent RAG for forensic evidential reasoning — structured domain harness reference design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CHARLIE is a fully on-premise multi-agent RAG system for digital forensic document analysis that orchestrates task decomposition, parallel subquery execution, structured memory, and validation within a LangGraph-based directed workflow. It was deployed operationally at Brazil's Federal District Forensic Institute, processing multi-year corpora of forensic reports under strict data-sovereignty and chain-of-custody constraints. The system demonstrates that agent-orchestrated, on-premise RAG architectures can replace single-pass RAG for large-scale evidential extraction without sacrificing auditability.

Problem

Classical single-pass RAG fails in forensic multi-document workflows: it cannot aggregate across large corpora, has no persistent intermediate state, is bounded by context-window size, and cannot decompose complex analytical tasks. Cloud-based LLM solutions are incompatible with forensic data-sovereignty, confidentiality, and chain-of-custody requirements. Existing agent frameworks assume open external API access, which is prohibited in regulated evidential environments.

Method

CHARLIE is built on LangGraph as a directed execution graph with deterministic, auditable node transitions covering: query classification → task planning → subquery decomposition → parallel RAG execution → structured memory update → self-reflection validation → synthesis. For a corpus of \(N\) documents and \(M\) target attributes, the system generates \(N \times M\) atomic subqueries, each independently processed through a two-stage retrieval pipeline (dense cosine retrieval, top-\(k=100\); cross-encoder reranking via ms-marco-MiniLM-L-6-v2, top-\(k=25\)). Results are stored in tabular or key-value structured memory with explicit provenance links before a final synthesis stage consolidates them. Inference runs on Qwen3-32B served via vLLM across four NVIDIA A40 GPUs with tensor parallelism (up to 131k context). Embeddings use nomic-embed-text via Ollama, fully local. The agent is structured around five components: Brain (LLM), Perception (retrieval), Action (constrained to internal ops only), Memory (short-term + persistent structured), and Self-Reflection (completeness and hallucination checks with optional re-execution).

Key Contributions

• A fully on-premise, open-source multi-agent RAG harness architected explicitly around forensic legal constraints (sovereignty, chain-of-custody, auditability).
• A \(N \times M\) subquery decomposition strategy enabling scalable corpus-level extraction independent of context-window limits.
• A LangGraph-based orchestration graph with deterministic, logged node transitions providing workflow-level traceability rather than relying on model introspection.
• A self-reflection validation layer that detects missing fields, structural inconsistencies, and hallucination signals before synthesis.
• Operational deployment with four years of longitudinal forensic intelligence generation (traffic accident analysis, femicide pattern analysis).

Results

• Traffic accident case study: Processed ~2,000 forensic reports per year; produced structured geospatial/statistical datasets supporting four consecutive years of annual reporting and longitudinal monitoring.
• Femicide pattern analysis: Extracted semantic patterns across five dimensions (instruments, modus operandi, victim/offender profiles, substance involvement) from heterogeneous narrative reports; enabled population-level pattern identification with full document traceability.
• No quantitative benchmark numbers are reported; evaluation is entirely qualitative/operational. Authors explicitly note future work will add quantitative metrics.

Limitations

• No quantitative evaluation metrics; results are case-study demonstrations without precision/recall or comparison to baselines.
• Datasets are confidential and cannot be released, limiting reproducibility verification.
• Hallucination risk is mitigated architecturally but not formally quantified.
• The \(N \times M\) subquery explosion may become computationally prohibitive at very large scale without further optimization.
• Self-reflection re-execution strategy is described but not formally evaluated for its effectiveness.
• Restricted to structured extraction tasks; complex inferential or causal reasoning across documents is out of scope.

Relevance to Harnesses / Meta-Harnesses

CHARLIE is a concrete instantiation of a domain-specific agent harness: it wraps an LLM within a deterministic, auditable multi-stage orchestration graph (LangGraph) that governs classification, decomposition, execution, memory, and validation — precisely the structure that harness/meta-harness research formalizes. The paper's design-principle-first approach (embedding sovereignty, traceability, and human-in-the-loop oversight directly into harness architecture) offers a case study in how harness-level constraints differ from task-level prompting. Its transition narrative from classical RAG to agent-orchestrated pipeline illustrates the practical motivation for harnesses: single-call LLMs cannot fulfill structured, multi-step analytical workflows, and the harness layer is what bridges that gap. For researchers tracking meta-harnesses, CHARLIE is notable for operating without any external tool calls — all actions are constrained to internal, pre-approved operations — demonstrating a closed-world harness design pattern that may generalize to other high-compliance domains.