Skip to content

AI translation of literary texts is "fine", but readers still prefer human translations

🕒 Published (v1): 2026-06-24 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper evaluates AI vs. human literary translation by recruiting 15 avid readers to read and compare ~8K-word excerpts for 15 novels (French, Polish, Japanese → English). Readers find MT "fine" but consistently prefer human translation, especially under close reading (522/772 chunk judgments favor HT), while automatic metrics including LLM-as-a-judge fail to recover human preferences and systematically favor MT.

Problem

Existing MT evaluation—automatic metrics, fluency/adequacy ratings, and segment-level annotation—fails to capture the holistic reading experience (immersion, voice, rhythm) that literary translation must preserve. There is no reader-centered evaluation framework for long-form literary MT, and it is unclear whether modern agentic LLM pipelines can match published human translations at the level readers actually care about.

Method

Human evaluation design: 15 avid readers (recruited via Upwork, 12–120 books/year) each evaluated two ~8K-word novel opening excerpts in two phases: - Immersive reading: each translation read in full and rated independently (acceptability, smoothness, immersion, willingness to continue), then compared head-to-head (30 comparisons total, counterbalanced order). - Close reading (after 24h break): 300-word HT–MT chunk pairs presented side-by-side; readers highlighted good/poor spans and stated preferences (772 comparisons across 386 aligned chunk pairs).

Agentic MT pipeline (P3, based on AutoFiction): Three-stage pipeline using Claude Code (Claude Opus 4.6) and Codex (GPT-5.4) as coding agents: 1. Preprocessing: Claude Code extracts register, terminology, named entities, and translation memory; text is divided into ~1K-token chunks preserving paragraphs. 2. Chunk-level loop: Codex translates each chunk; parallel reviews check (a) translation quality/faithfulness (modified LiTransProQA, Codex) and (b) literary quality/voice (Claude Code). An acceptance gate cycles failed chunks through targeted revision up to 3× local cycles; if 3 consecutive revisions fail, the model switches. 3. Excerpt-level loop: Revised chunks are merged; Codex runs global literary quality and cross-chunk continuity reviews; Claude Code revises if the final acceptance gate is not passed (up to 2× global cycles).

Pipeline was selected via a blind 5-way preference study on a 16-book dev set; P3 was chosen most often (5/16 first-choice votes) while P1 with GPT-5.4 was a clear outlier (1/16).

Key Contributions

  • LAIT dataset: 15 FR/PL/JA→EN novel openings (2025–2026) with source, published HT, and agentic MT aligned at excerpt, chunk, and paragraph levels; 2K ratings/preferences, 1K comments, 7.2K span-level annotations; plus a 16-book unannotated dev set with 1.7K paragraph-level alignments across 5 MT systems.
  • Reader-centered evaluation protocol combining immersive and close reading with span-level annotation, released with guidelines and annotation software.
  • Empirical findings on modern literary MT quality, within-book variance, reader detectability, and the failure of automatic metrics.

Results

  • Excerpt-level (immersive): 19/30 judgments favor HT; order-adjusted HT probability = 63.4% (p=.148, not significant). 54% of MT responses showed moderate-to-strong willingness to continue reading vs. 66% for HT.
  • Chunk-level (close reading): 522/772 judgments favor HT (67.5%), including 205 strong HT preferences vs. 54 strong MT preferences; order-adjusted HT probability = 76.2% (p=.011, significant).
  • Ratings: HT had 4.0× the odds of a higher acceptability rating (p=.0069) and 4.3× the odds of a higher smoothness rating (p=.0029) vs. MT.
  • Span annotations: HT received 107.8 positive-highlighted words/1K vs. 68.5 for MT (rate ratio = 1.69, p<.001); MT received 100.7 negative-highlighted words/1K vs. 42.9 for HT (rate ratio = 0.43, p<.001).
  • Detectability: Readers identified MT correctly only 17/30 times (≈chance); they were misled by surface "AI tells" (em-dashes, absence of swear words) and showed a tendency to prefer whichever version they believed was human.
  • MT variance: MT quality varies significantly more within a single book than HT; book identity (not source language) strongly predicts chunk-level preference (p<.001). One title (Hooked) showed 88% MT preference, suggesting HT is not universally superior.
  • Automatic metrics: LLM-as-a-judge and other automatic metrics fail to recover reader preferences and systematically favor MT.

Limitations

  • Small sample: 15 readers, 15 books, one agentic pipeline; statistical power is low at excerpt level (p=.148).
  • All translations target English; findings may not generalize to other target languages where LLMs are weaker.
  • Only one MT pipeline (P3) evaluated in the main human study; pipeline selection margin was small and not decisive.
  • Pipeline cost (\(400 USD) and evaluation cost (\)4K USD) limit scale and reproducibility.
  • No reader expert in literary translation; readers were avid but not professional critics or translators.
  • Results on translations into Japanese, Polish, French, and Spanish are described as "preliminary" and only partially reported in the excerpt provided.

Relevance to Agentic AI / LLM Agents

This paper is a direct case study of an agentic LLM pipeline deployed at production-relevant quality thresholds: a two-agent system (Claude Code + Codex) with multi-stage review loops, acceptance gates, model-switching on repeated failure, and style-guided generation over long contexts (~8K words). It demonstrates that multi-agent architectures can reach near-human performance on a hard creative task while exposing a key gap: the agentic pipeline's output is harder to evaluate automatically—LLM-as-a-judge metrics diverge from human preference in the direction opposite to ground truth. This finding directly challenges the use of LLM judges as proxies for human evaluation in agentic creative systems, a widely assumed shortcut in the agent evaluation literature.