Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery¶

🕒 Published (v1): 2026-06-18 05:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper addresses "trajectory collapse" in LLM-based legal e-discovery—where early misclassifications silently compound across multi-step reasoning chains to invalidate entire privilege reviews. The authors propose a four-layer Human-on-the-Loop (HOTL) orchestration framework with calibrated uncertainty-gated escalation. A preliminary simulation on 5,000 synthetic documents shows up to 61% reduction in privilege-waiver risk while routing fewer than 25% of documents to attorney review.

Problem¶

Agentic ReAct-style workflows for electronic discovery condition each step on prior outputs, so a single misclassification propagates silently through hundreds of downstream decisions (trajectory collapse). Conventional endpoint metrics (precision/recall/F1) do not detect this because they measure only final outputs, not trajectory integrity. RLHF-trained LLMs exacerbate this via the Fluency Trap (confident fluent justifications for wrong legal classifications) and hallucinated tool calls.

Method¶

A four-layer verification architecture intercepts failures at each functional stage:

Planning Validation: A fine-tuned DeBERTa-v3 classifier \(f_{plan}\) estimates solvability \(P(S | q, T)\) for query \(q\) over toolset \(T\); queries below \(\tau_{plan} = 0.70\) trigger HOTL escalation before execution begins. Calibrated to ECE = 0.04.
Stepwise Reasoning Checkpoints: At each step \(t\), an advantage-style attribution score \(A_t = \mathbb{E}_\pi[R(\tau) | s_t, a_t] - V(s_t)\) is computed via \(k{=}20\) Monte Carlo rollouts using GPT-4o itself as the value approximator. If \(A_t < \epsilon = 0.05\), the trace is discarded and the agent resamples from \(s_t\).
Execution Sandboxing: Write actions are executed in a dry-run environment; \(\text{Commit}(a) \Leftrightarrow \text{FRCPCompliant}(S \oplus \delta(a))\) enforced by a deny-list policy engine. Irreversible API calls use a compensating-transaction pattern for rollback.
Uncertainty-Gated Escalation: Total predictive uncertainty is decomposed as \(U_{total} = \underbrace{\mathbb{E}_{p(\theta|D)}[H(Y|X,\theta)]}_{\text{Aleatoric}} + \underbrace{I(Y;\theta|X,D)}_{\text{Epistemic}}\). Epistemic uncertainty \(U_{ep}\) is approximated via self-consistency variance across \(k{=}10\) independent rationale samples (SelfCheckGPT-style). When \(U_{ep} > \tau_{esc}\), the system suspends and surfaces trajectory state to an attorney. Calibration: ECE = 0.06, Brier = 0.11.

Key Contributions¶

Taxonomy of agentic failure modes in legal IR organized by functional stage (planning, retrieval, reasoning, execution), with FRCP-specific regulatory consequences mapped to each failure class.
Four-layer HOTL verification architecture with stage-targeted interventions (planning validator, reasoning checkpoints, FRCP sandbox, uncertainty-gated escalation).
New trajectory-integrity metrics: First-Error Position (FEP) and Rollback Recovery Rate (RRR), complementing endpoint metrics.
Preliminary simulation demonstrating the PWR vs. escalation-rate tradeoff across three \(\tau_{esc}\) thresholds.

Results¶

On a synthetic 5,000-document corpus with ground-truth attorney labels:

Condition	PWR (95% CI)	Escalation Rate	Precision	Recall
Autonomous	8.3% [7.1, 9.5]	0%	0.81	0.74
T-HOTL τ=0.7	5.1% [4.2, 6.1]	12.4%	0.87	0.82
T-HOTL τ=0.5	3.2% [2.5, 3.9]	23.7%	0.91	0.88
T-HOTL τ=0.3	1.7% [1.2, 2.3]	47.9%	0.94	0.92
Manual	0.4% [0.1, 0.8]	100%	0.97	0.96

T-HOTL at \(\tau{=}0.5\) achieves 61% PWR reduction vs. autonomous (non-overlapping 95% CIs) while escalating only 23.7% of documents.
Mean RRR = 0.71 across all T-HOTL conditions (reasoning checkpoints recovered 71% of intercepted error trajectories without human escalation).
Mean FEP = 3.2 steps (out of 7.8 average trajectory length) in the autonomous condition—errors originate early and compound substantially.
Processing latency: 14.2 s/document at \(\tau{=}0.5\) with fully parallelized batched API calls; sequential execution would require 290–340 s.

Limitations¶

Corpus is entirely synthetic; distributional complexity of production data (domain jargon, encoded attachments, multi-party privilege chains) is unmodeled.
Ground-truth labels lack formal inter-annotator agreement (Cohen's \(\kappa\) not reported across privilege subclasses).
Individual layer contributions are not ablated; each layer's independent impact on PWR and FEP is unknown.
GPT-4o raises data-governance concerns for privileged documents; zero-egress/on-premises inference required for production deployment.
\(k{=}20\) MC rollouts for reasoning checkpoints and \(k{=}10\) self-consistency samples make sequential inference impractically slow (290–340 s/doc); batched inference is a hard prerequisite.
Value function \(V(s_t)\) is approximated by MC rollouts using the same model being evaluated—a potentially circular estimator.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a concrete instantiation of a domain-specific meta-harness: a multi-layer orchestration wrapper that intercepts and reroutes an underlying agent's execution at planning, reasoning, execution, and output stages. The four-layer architecture exemplifies the canonical harness pattern—wrapping an opaque base agent (GPT-4o ReAct) with stage-specific validators, sandboxes, and escalation logic—and introduces trajectory-integrity metrics (FEP, RRR) that are directly applicable for monitoring harness effectiveness beyond endpoint accuracy. The HOTL escalation mechanism is a formalized version of the human-in-the-loop interrupt that meta-harnesses must implement to safely delegate to autonomous agents in high-stakes settings. The uncertainty-gated escalation design and compensating-transaction rollback pattern are directly reusable components for harness builders operating in any high-consequence agentic pipeline.