Skip to content

RAGthoven at SemEval-2026 Task 1: A Multi-Stage Pipeline Walks Into a Benchmark and Barely Clears the Bar

🕒 Published (v1): 2026-07-14 18:39 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RAGthoven is a theory-grounded, four-stage LLM pipeline (Planner → Best-of-N Writer → Reflector → Judge) for constrained multilingual humor generation, augmented with RAG over a 98-joke corpus. It ties Rank 1 with the Gemini 2.5 Flash single-prompt baseline in all three languages at SemEval-2026, despite far greater computational cost. Two agentic variants (ReAct sequential and autonomous multi-branch) failed to outperform the non-agentic pipeline on a held-out sample.

Problem

LLMs recycle a narrow set of joke patterns and struggle with genuine novelty; constrained humor generation (verbatim keyword inclusion, headline reference, length caps, multilingual) further narrows the effective output space. Existing single-prompt approaches lack grounding in computational humor theory and provide no mechanism to diversify the joke-mechanism space at ideation time.

Method

The pipeline decomposes generation into four prompted LLM stages, each grounded in humor theory (SSTH, BVT):

  1. Planner — given input constraints plus (in EXP08) top-4 jokes retrieved from a 98-entry corpus via all-MiniLM-L6-v2 embedding + ms-marco-MiniLM-L-12-v2 cross-encoder reranking, produces a structured plan: premise, two juxtaposed scripts, benign-violation angle, anchor tokens, and punchline mechanism.
  2. Writer — generates \(N \in \{4, 10, 12\}\) candidate jokes (best-of-N sampling), enforcing hard constraints inline (verbatim word inclusion, length cap, no semicolons, no cliché openers).
  3. Reflector — diagnoses failure modes across the candidate set (e.g., "punchline predictable," "word inclusion forced") and emits 1–2 revised candidates.
  4. Judge — scores all surviving candidates on a 5-criterion rubric (surprise/resolution, benign violation, specificity, punchiness, constraint compliance) and selects the best.

EXP09 reimplements these four stages as ReAct-style sequential tool-calling subagents plus a deterministic CONSTRAINTAUDIT checker (up to 24 iterations). EXP10 adds autonomous multi-branch exploration with dynamic tool ordering and 2–4 parallel branches (up to 36 iterations). The system is built on the RAGthoven configuration-driven framework; no model weights are modified.

Key Contributions

  • Four-stage humor-theory-grounded LLM pipeline with inline constraint enforcement and rubric-driven selection.
  • RAG-augmented ideation (EXP08): semantic retrieval over a mechanism-annotated joke corpus diversifies the Planner's creative frame.
  • Ten-experiment ablation tracing which stage additions drive qualitative improvement.
  • Negative result: across four frontier models, neither ReAct sequential (EXP09) nor autonomous multi-branch agentic orchestration (EXP10) yielded outputs judged superior to the non-agentic pipeline, despite up to \(3\times\) more tool-call budget.
  • Rank 1 (tied) at SemEval-2026 MWAHAHA Subtask A in English, Spanish, and Chinese using identical language-agnostic prompts.

Results

  • Spanish: Elo 1182 (RAGthoven) vs. 1140 (Gemini 2.5 Flash baseline); +42 raw Elo, both Rank 1 (statistically tied).
  • English: Elo 1045 vs. 1081 (baseline); within a 9-system tied top group spanning ~47 Elo points.
  • Chinese: Elo 1045 vs. 1053 (baseline); within an 8-system tied top group.
  • Qualitative ablation identifies two major inflection points: best-of-12 sampling (EXP04) and RAG-augmented planning (EXP08); Reflector (EXP07) provides targeted rescue for "forced" word-inclusion failures.
  • EXP09/10 agentic variants: no consistent quality advantage over non-agentic GPT-5 (EXP08) on a 12-instance held-out English sample across all four model variants (GPT-5, Gemini 3 Pro, Claude Sonnet 4.5, Claude Opus 4.5).

Limitations

  • No cost–quality study: token counts and wall-clock latency per instance were not instrumented; EXP10 branches were frequently discarded, inflating tool-call counts with no measurable quality gain.
  • English-centric RAG corpus: 98 jokes, English-annotated mechanisms; no cross-lingual retrieval signal beyond base-model multilingual capacity.
  • Language-agnostic rubric: Judge and Reflector prompts apply a single English-defined rubric to all three languages, potentially biasing toward English humor conventions.
  • No stage-level ablation: which single stage is the binding quality constraint is not isolable from the reported results.
  • Reproducibility: results depend on proprietary, version-drifting model APIs.
  • Small agentic evaluation: negative finding on EXP09/10 is based on a 12-instance held-out sample with manual (not Elo-ranked) evaluation.

Relevance to Harnesses / Meta-Harnesses

RAGthoven is a direct instance of a configuration-driven LLM meta-harness: stages (Planner, Writer, Reflector, Judge) are modular prompt templates wired together by an orchestrator, with a deterministic CONSTRAINTAUDIT tool injected as a feedback loop — the same architectural pattern as agent scaffolds like ReAct and Reflexion, but applied to creative generation. The paper's central negative result — that ReAct-style sequential tool-calling and autonomous multi-branch orchestration both failed to improve over the fixed-topology non-agentic pipeline — is a direct empirical data point for the diminishing-returns question in harness design: increased orchestration complexity does not reliably outperform a well-tuned static pipeline when a strong frontier model underlies each stage. The ten-experiment ablation also provides a rare public trace of iterative harness development, making it a useful reference for how prompt-stage decomposition and retrieval integration interact across 10 configuration steps.