TransLaw: A Large-Scale Dataset and Multi-Agent Benchmark Simulating Professional Translation of Hong Kong Case Law¶

🕒 Published (v1): 2025-07-01 15:39 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

TransLaw is a 7-agent multi-agent framework that decomposes English-to-Chinese Hong Kong Court of Final Appeal judgment translation into command, execution, and review modules, augmented with RAG over a legal glossary and iterative feedback. Alongside the framework, the authors release HKCFA Judgement 97-22, the first sentence-level parallel corpus of HK case law (11,099 pairs, 1997–2022). Benchmarking 13 LLMs shows TransLaw substantially outperforms single-agent translation on automated metrics, though it still trails professional human translators.

Problem¶

Manual translation of HKCFA judgments is costly and inconsistent; single-LLM approaches hallucinate legal terms, miss socio-cultural nuance, and violate the strict formatting conventions of the Court of Final Appeal. No public bilingual HK case law dataset exists, leaving both MT and LLM-based legal translation benchmarks sparse for this jurisdiction.

Method¶

TransLaw defines a collective $\mathcal{C} = \{A_{Com}, A_{Term}, A_{Trans}, A_{Align}, A_{TermR}, A_{Cita}, A_{StyleP}\}$ organized into three modules:

Translation Command Module ($M_{Com}$): $A_{Com}$ segments source judgment $J = \{s_i\}_{i=1}^N$, dispatches sentences, and runs an iterative refinement loop: $$\hat{s}_i^{(k+1)} = \hat{s}_i^{(k)} \oplus \Psi(F_i^{(k)})$$ terminating when $F_i^{(k)} = \emptyset$ or iteration threshold $K$ is reached. A memory $E_i = \{\hat{s}_1, \ldots, \hat{s}_{i-1}\}$ provides inter-sentence coherence.
Translation Execution Module ($M_{Exec}$): $A_{Term}$ retrieves candidate expressions for legal terms from the DOJ Combined Glossaries via RAG ($C_k = R_D(l_k)$); $A_{Trans}$ injects verified term set $\hat{L}_i$ into the sentence and adjusts for global context $G$: $\hat{s}_i^{(0)} = f_{Trans}(s_i, \hat{L}_i, G)$.
Expert Review Module ($M_{Rev}$): four parallel reviewers generate feedback $\delta_i^{(k)} = \delta_{Align}^{(k)} \cup \delta_{TermR}^{(k)} \cup \delta_{Cita}^{(k)} \cup \delta_{Style}^{(k)}$, covering semantic alignment, glossary compliance, citation formatting, and judicial tone.

Human evaluation uses the proposed Legal ACS metric: $I = \alpha A + \beta C + \gamma S$ with $\alpha=0.6$, $\beta=0.3$, $\gamma=0.1$.

Key Contributions¶

First systematic benchmark of 13 open-source and commercial LLMs on English-to-Chinese HK case law translation.
HKCFA Judgement 97-22 dataset: 344 professionally translated judgments, 11,099 sentence pairs, 811k English tokens and 1.3M Chinese tokens, with structure-based alignment from official HTML.
TransLaw multi-agent pipeline with RAG-backed legal glossary, central orchestrator, and multi-dimensional iterative review.
Legal ACS metric with weighted accuracy/coherence/style dimensions and professional-translator human evaluation on 200 paragraph pairs.

Results¶

TransLaw + GPT-4o: xCOMET-XL 85.12 vs. single-agent 69.42; wmt22-unite-da 91.78 vs. 75.88 — a ~16-point absolute gain on xCOMET-XL.
TransLaw improves over the single-agent baseline across all 13 tested LLMs.
GPT-4o ranks first in both TransLaw and single-agent settings; open-source models slightly trail closed-source but all benefit from the multi-agent scaffold.
Instruction-tuned chat variants (e.g., Baichuan-13B-Chat) outperform base models (Baichuan-13B-Base), confirming alignment fine-tuning aids translation.
Legal-specific LLMs (ChatLaw-13B/33B) rank 12th and 11th — underperforming general LLMs of comparable scale.
Human evaluation: TransLaw outperforms single-agent GPT-4o but trails official human translation, particularly on contextualizing complex terminology and stylistic naturalness.

Limitations¶

Still below professional human translators on ACS dimensions of complex-term contextualization and stylistic naturalness.
Legal-specific LLMs do not outperform general ones; authors attribute this to weak base models and limited benefit from legal fine-tuning, but do not fully resolve the cause.
Human evaluation covers only one judgment (FACC 1/2021, 200 paragraphs), limiting generalizability of ACS scores.
Computational cost of 7 sequential/iterative LLM calls per sentence is not analyzed or reported.
Dataset covers 1997–2022; domain shift for post-2022 judgments is unaddressed.

Relevance to Harnesses / Meta-Harnesses¶

TransLaw is a domain-specialized multi-agent harness with a textbook meta-harness structure: $A_{Com}$ acts as the orchestrator that dispatches work to specialist sub-agents, collects structured feedback, and drives an iterative refinement loop — the same command/execution/review topology that recurs in general-purpose harness designs. The iterative feedback equation ($\hat{s}_i^{(k+1)} = \hat{s}_i^{(k)} \oplus \Psi(F_i^{(k)})$) is a concrete instantiation of the "loop until convergence" pattern that meta-harnesses often abstract as a generic retry or reflection primitive. The paper's ablation across 13 backbone LLMs — with the same harness topology but swapped agents — directly quantifies how harness design decouples from model choice, an empirical data point useful for harness researchers studying modularity. It also demonstrates that role specialization (dedicated citation agent, dedicated terminology agent) produces measurable quality gains over a monolithic single-agent baseline, supporting the case for fine-grained task decomposition in harness construction.