Skip to content

TxSum: User-Centered Ethereum Transaction Understanding with Micro-Level Semantic Grounding

🕒 Published (v1): 2025-12-07 17:23 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TxSum defines a new NLP task and 187-transaction annotated dataset for user-centered Ethereum transaction explanation, pairing natural-language summaries with five-attribute token-flow semantic labels. MATEX, the accompanying multi-agent framework, uses uncertainty-triggered retrieval (DKIP) and adversarial trace-based auditing (AGCP) to generate grounded, factually consistent DeFi explanations. On complex transactions, MATEX raises user comprehension from 52.9% to 76.5% and malicious-transaction rejection from 36% to 88%.

Problem

Modern DeFi transactions are compositionally complex, yet existing tools (MetaSuites, EigenPhi, Tenderly) expose only raw syntactic traces without economic semantics, causing widespread "blind signing." No structured NLP task or grounded explanation method existed for translating on-chain call traces into user-facing, risk-aware narratives verified against the raw trace.

Method

MATEX implements a dual-process (System 1 / System 2) multi-agent workflow orchestrated with LangGraph:

  1. Profiler (System 1): Scans all token flows \(t_i\) and computes a semantic uncertainty score \(u_i\) per flow.
  2. Dynamic Knowledge Injection Protocol (DKIP): If \(u_i > \tau\), escalates to System 2 and triggers the Investigator; otherwise, the lightweight explanation path proceeds. This avoids blanket retrieval.
  3. Investigator (System 2): Retrieves targeted protocol evidence (ABI entries, verified docs) only for uncertain flows.
  4. Synthesizer: Fuses on-chain evidence, retrieved context, and flow-level hypotheses into a draft 3–4 sentence summary.
  5. Safety Auditor + Auditor-Guided Context Pruning (AGCP): Checks the draft against the raw trace; identifies error-causing context \(P_{\text{mask}}\) and removes it from the working context before re-drafting, suppressing repeated hallucinations.

All agents share the same Qwen3-Max backbone; the VLM baseline uses Qwen3-VL-Plus for multimodal input. The dataset (187 transactions, avg. 12.7 token-flow steps) annotates five per-flow attributes: action_type (14-class), intent, mechanism, precondition, result. Inter-annotator agreement: Cohen's \(\kappa = 0.98\) for action_type; BERTScore F1 \(> 0.91\) for all NL fields.

Key Contributions

  • Semi-structured interviews with 16 Web3 users identifying three explanation requirements: structured, risk-aware, token-flow-grounded.
  • TxSum task formulation: two-level output (transaction summary + token-flow semantic labels with five attributes) from strictly on-chain observable inputs.
  • TxSum dataset: 187 annotated complex Ethereum transactions across five DeFi categories, with verified inter-annotator reliability.
  • MATEX framework: DKIP for selective uncertainty-triggered retrieval and AGCP for adversarial context pruning, realized as four specialized agents in LangGraph.
  • Multi-faceted evaluation: automatic metrics, LLM-as-judge (GPT-5.1), DeFi expert annotation, controlled user comprehension study (N=20), and security risk study.

Results

  • EO1 (Explanation Quality) — MATEX vs. best baseline (Single-Agent), LLM-as-judge / Human:
  • F1 (token-flow faithfulness): 73.2% / 75.1% vs. 67.6% / 69.0%
  • F2 (intent quality): 76.6% / 78.3% vs. 67.5% / 69.2%
  • T2 (clarity): 93.0% / 94.0% vs. 89.8% / 88.1%
  • T1 (outcome correctness): 73.3% / 76.0% (MATEX highest among humans)
  • Also best on ROUGE-L, BLEU, Exact Match, action_type Accuracy, Macro-F1
  • EO2 (User Comprehension) — MATEX vs. Single-Agent vs. Raw on complex transactions:
  • Comprehension accuracy: 76.5% vs. 52.9% vs. 35.3%
  • Perceived trust/clarity (1–5 scale): 3.7 vs. 2.6 vs. 1.8
  • EO3 (Risk Awareness) — MATEX vs. Raw (N=10 per group):
  • Malicious transaction rejection: 88% vs. 36% (\(p = 0.002\))
  • Correct threat identification: 72% vs. 28%
  • False-rejection on benign transactions: 8% vs. 12% (difference not significant, \(p = 0.741\))
  • Ablation: Removing the Investigator causes the largest single degradation (F1: \(73.2 \to 62.8\)); removing the Safety Auditor collapses micro-level faithfulness (F1: \(73.2 \to 62.5\)) while barely affecting outcome correctness.

Limitations

  • Dataset size is small (187 transactions); coverage of adversarial/novel DeFi primitives may be incomplete.
  • Annotation restricted to transactions with 5–30 token flows, excluding very short (trivial) and very long (annotation-infeasible) cases.
  • Qwen3-Max backbone is proprietary; reproducibility with open-weight models is not assessed.
  • User studies use small cohorts (16 interviews, 20 comprehension participants) recruited from Web3 communities, limiting generalizability.
  • AGCP removes error-causing context but does not guarantee convergence; the number of auditing iterations and failure modes under highly novel protocols are not fully characterized.
  • The five semantic attributes were derived from a single interview study; their completeness for all DeFi transaction types is not formally validated.

Relevance to Harnesses / Meta-Harnesses

MATEX is a domain-specific multi-agent harness that assigns distinct cognitive roles (Profiler, Investigator, Synthesizer, Safety Auditor) to specialized agents and orchestrates them with LangGraph, making it a concrete instantiation of role-specialized harness architecture. Its two control mechanisms—DKIP (selective escalation gated by a per-item uncertainty score) and AGCP (adversarial context pruning that modifies the shared working state before re-generation)—are directly transferable harness-design patterns for any pipeline where hallucination risk is heterogeneous across inputs and iterative refinement must avoid error accumulation. The ablation results provide empirical evidence for the value of each harness component in isolation, which is useful for harness builders calibrating architecture complexity against task requirements. The LangGraph orchestration layer, shared tool access contracts, and stateful inter-agent communication exemplify the engineering substrate that meta-harness designers need to reason about when composing reusable agent pipelines.