Skip to content

Building from Scratch: A Multi-Agent Framework with Human-in-the-Loop for Multilingual Legal Terminology Mapping

🕒 Published (v1): 2025-12-15 03:29 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a multi-agent, human-in-the-loop (HITL) workflow for constructing a trilingual (Chinese-English-Japanese) legal terminology database from scratch. Specialized LLM agents handle OCR, article alignment, term extraction, and QA while domain experts provide oversight and corrective feedback at every stage. The framework outperforms traditional manual and automated methods in precision, consistency, and scalability.

Problem

Multilingual legal terminology mapping between Chinese and Japanese is severely under-resourced; existing tools fail because the languages share many homographs with divergent legal meanings (e.g., "特許" means "patent" in Japanese but "special authorization" in Chinese). Prior approaches—SMT, NMT, and raw LLM prompting—lack end-to-end quality assurance and systematic update mechanisms for this language pair and legal domain.

Method

A five-stage pipeline, each stage implemented as a team of specialized LLM agents with embedded human checkpoints:

  1. Preprocessing: Four coordinated agents (PDF Parsing, Style QC, Content Segmentation, Segmentation QC) convert PDFs → clean article-level JSON segments. GPT-4.1-mini handles OCR; DeepSeek-v3/r1 handles segmentation and error reprocessing.
  2. Article alignment: A Bilingual Article Aligning Agent combines rule-based matching (regex patterns like "第X条") with embedding-based cosine similarity (OpenAI text-embedding), followed by Jina/BGE-m3 reranking to select best-match articles; a QC agent flags failures for human re-review. Bilingual pairs are then extended to trilingual by adding English as a semantic pivot.
  3. Term extraction & mapping: Bilingual Term Extraction, Auto-Complete, and Trilingual Mapping agents extract candidate term pairs with few-shot prompting and produce JSON-structured trilingual mappings.
  4. Standardization: Entries are normalized against ISO/TBX standards.
  5. Quality assurance: A five-dimensional evaluation framework (Coverage, Consistency, Completeness, Professionalism, Translation Quality) with 17 sub-criteria applicable to both human and machine assessment.

Human experts (senior translators, legal experts, terminologists, language revisers) are embedded at every stage, providing targeted corrective feedback fed back into agent prompts. The platform is deployed as a cloud-native "Terminology-as-a-Service" (TAAS) architecture with CI/CD pipelines for continuous updates.

Key Contributions

  • A large-scale article-aligned trilingual legal parallel corpus (35 Chinese statutes; 5,172 article-level segments per language, ~1M total words).
  • A reproducible multi-stage multi-agent pipeline for cross-lingual legal terminology extraction with systematic HITL validation.
  • A unified five-dimensional, 17-sub-criteria quality framework applicable to both human reviewers and automated scoring.
  • An open, AI-compatible TAAS platform with LLM-friendly APIs, collaborative editing, and CI/CD update support.

Results

  • 5,172 article-level aligned segments per language; Chinese avg 42.0 words, English 70.0, Japanese 75.1.
  • Alignment process achieves 100% article-level alignment success rate (failures trigger automatic retry + human review loop).
  • Human-in-the-loop multi-agent workflow yields improved precision and consistency over traditional manual methods and standard automated pipelines (specific F1/precision figures not provided in available text).
  • Several open-source LLMs (not named in available text) matched proprietary models on legal term extraction, demonstrating cost-effective alternatives.

Limitations

  • Quantitative comparison numbers (F1, precision, recall vs. baselines) are not fully reported in the provided text.
  • Corpus is limited to 35 Chinese statutes; coverage of niche legal subdomains may be incomplete.
  • Requires sustained engagement from 2–3 human experts at every stage, constraining throughput and scalability to organizations with domain expert access.
  • English serves as pivot, which may introduce semantic drift for concepts without clear English equivalents.
  • The termbase is still being made fully publicly accessible ("in the near future").

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete domain application of the multi-agent harness pattern: a meta-harness orchestrates a sequential pipeline of specialized sub-agents, each with its own QC agent, while a human-in-the-loop layer provides supervisory control across all stages. The feedback injection mechanism—where human reviewers return structured corrections to downstream agents as updated prompts—is a practical HITL harness design that is directly portable to other pipeline architectures. The TAAS + CI/CD deployment model mirrors the continuous-delivery harness pattern, showing how agent pipelines can be made operationally sustainable at production scale. The explicit role separation between automation (repetitive extraction, OCR, alignment) and human judgment (legal nuance, final validation) is a generalizable blueprint for harness design in expert-knowledge domains.