Skip to content

MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference

🕒 Published (v1): 2025-09-26 07:31 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MARCH is a 2,209-question benchmark targeting the underexplored intersection of multi-hop reasoning and layered ambiguity, where ambiguity at early hops creates latent uncertainty that cascades downstream. The authors also propose CLARION, a two-stage agentic framework that decouples ambiguity planning from evidence retrieval, consistently outperforming all baselines across three LLMs.

Problem

Existing QA benchmarks address either multi-hop reasoning (MuSiQue) or single-hop ambiguity (ASQA) in isolation. Real-world queries frequently combine both: a polysemous term at hop 1 can make the ambiguity of hop 2 latent and only discoverable if all interpretations of hop 1 are preserved. Standard LLMs prematurely commit to one reading, causing cascading failures. No benchmark existed to measure this compound difficulty, and a 13.3% overlap rate in the lmsys-chat-1m corpus confirms it is practically common.

Method

Dataset construction (MARCH): Starting from MuSiQue, a four-stage pipeline filters and enriches questions: (1) multi-LLM ambiguity detection requiring full agreement across four models (gpt-4.1, llama-4-maverick, qwen3-235b, claude-sonnet-4) under the rule \(\varphi_t(q) = \mathbb{I}\!\left(\frac{1}{|M|}\sum_{m\in M} y_{m,t}(q)=1\right)\); (2) Wikipedia retrieval via atomic sub-question decomposition + Qwen3-8B-Embedding re-ranking; (3) per-interpretation short answers and a synthesized long answer generated by gpt-4.1; (4) three-model unanimous-alignment filtering plus short-answer deduplication, yielding 2,209 examples across three ambiguity types (semantic/syntactic/constraint).

CLARION framework: A Planning Agent first detects ambiguity, classifies its type, and rewrites the query into clarified variants (one per valid interpretation). These variants are passed to an Acting Agent running a ReAct-style Thought→Action→Observation loop with three possible actions: Search (retrieve docs), Planning (self-correct the interpretation set mid-loop), and Answer (synthesize covering all interpretations). Execution is capped at five iterations. All actions are JSON-formatted for reliable parsing.

Key Contributions

  • MARCH benchmark: 2,209 multi-hop ambiguous questions with type labels (semantic/syntactic/constraint), clarified questions, per-interpretation short answers, supporting passages, and synthesized long answers; human-validated with Fleiss' \(\kappa\) up to 0.95.
  • Three-type taxonomy of multi-hop ambiguity (Entity-Driven/Structure-Driven/Scope-Driven), extending prior single-hop frameworks.
  • Multi-LLM consensus detection protocol (4-model full-agreement gate) to reduce single-model labeling bias.
  • CLARION, a two-stage agentic framework (Planning Agent + Acting Agent with mid-loop self-correction via re-invocable Planning action) that explicitly prevents premature branch pruning.
  • Empirical demonstration that SOTA models fail at the compound task even when they handle each sub-task adequately in isolation.

Results

  • CLARION (Qwen3-235b): STR-EM 38.73, Disambig-F1 28.38, Avg 33.56, LLM-Judge 3.474 — best across all models and metrics.
  • CLARION (DeepSeek-v3.1): Avg 29.25 vs. best baseline ReAct 23.97 (+5.3 pts).
  • CLARION (Gemini-2.5): Avg 27.71 vs. best baseline CoT w/ RAG 25.23 (+2.5 pts).
  • Human performance: STR-EM 73.00, Disambig-F1 62.00, Avg 67.50 — large gap above all models.
  • Disambig-F1 shows the largest gains for CLARION, confirming it reduces the "missing branch" failure mode that other methods do not address.
  • Ablating clarification (-w/o clarification) drops Qwen3 Avg from 33.56 to 25.33; further removing detection drops it to 23.48.
  • MuSiQue and ASQA suffer substantial performance drops when multi-hop and ambiguity intersect (up to −13.0 points shown in Figure 2).

Limitations

  • MARCH is derived exclusively from MuSiQue's validation/training sets, limiting domain diversity beyond what MuSiQue already covers.
  • The ReAct loop is capped at five iterations; unresolved cases are forced to answer, potentially underestimating CLARION's ceiling.
  • All retrieved evidence is from English Wikipedia only, excluding multilingual and heterogeneous sources.
  • The human performance gap (~67.5 avg vs. ~33.56 best model) is large but measured on only 60 annotated samples (20 per type), not the full benchmark.
  • LLM-as-a-Judge metric uses GPT-4, introducing potential evaluation bias toward GPT-family outputs.

Relevance to Harnesses / Meta-Harnesses

CLARION is a two-stage agentic harness where a Planning Agent orchestrates downstream Acting Agents: the Planning Agent emits a structured execution plan (clarified interpretation variants), and the Acting Agent consumes that plan while retaining the ability to re-invoke the planner mid-run as a self-correction action — a form of meta-level control flow within the harness. The dataset construction pipeline itself is a four-stage multi-LLM harness that chains detection, retrieval, generation, and filtering agents with a consensus-gate quality control layer, directly analogous to multi-agent meta-harness architectures for data quality. For researchers tracking harness design, CLARION's "Planning-before-Acting" separation and in-loop planner re-invocation offer a concrete, benchmarked pattern for managing latent branching in agentic reasoning pipelines.