Theorem-of-Thought: A Multi-Agent Framework for Abductive, Deductive, and Inductive Reasoning in Language Models¶

🕒 Published (v1): 2025-06-08 12:28 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Theorem-of-Thought (ToTh) decomposes LLM reasoning into three parallel specialized agents—abductive, deductive, and inductive—each producing a Formal Reasoning Graph (FRG), then selects the most coherent graph via NLI-calibrated Bayesian belief propagation. On symbolic (WebOfLies) and numerical (MultiArith) benchmarks, ToTh outperforms CoT, Self-Consistency, and CoT-Decoding across Mistral-7B, DeepSeek-7B, and Phi-3.5 Mini. The framework runs in $O(k \cdot s)$ time, linear in agents $k$ and reasoning steps $s$.

Problem¶

Existing LLM reasoning methods (CoT, Self-Consistency, CoT-Decoding) produce linear or stochastic reasoning traces with no mechanism to verify internal logical coherence. They conflate abductive, deductive, and inductive inference into a single undifferentiated pass, yielding outputs that are fluent but logically unsound and difficult to interpret.

Method¶

ToTh deploys three independent solver agents over input question $q$: - Abductive agent $a_1$: $\arg\max_H P(H \mid O, K)$ - Deductive agent $a_2$: $\{P_1, \ldots, P_n\} \vdash C$ - Inductive agent $a_3$: $\{x_1, \ldots, x_n\} \Rightarrow R$

Each agent's reasoning trace is converted into a Formal Reasoning Graph (FRG) $G^{(i)} = (V^{(i)}, E^{(i)})$, where nodes are intermediate conclusions and directed edges represent inferential steps scored by a pretrained NLI model (RoBERTa-MNLI): entailment $\theta=0.95$, neutral $\theta=0.60$, contradiction $\theta=0.10$.

Bayesian belief propagation over each FRG updates per-node confidence via: $$P(v_c) = \frac{P(v_p) \cdot \theta_{pc}}{P(v_p) \cdot \theta_{pc} + (1-P(v_p))(1-\theta_{pc})}$$ with prior $P(v)=0.5$ and multi-parent averaging. Each graph is scored as $\text{Score}(G^{(i)}) = \mu^{(i)} - H^{(i)}$ (mean confidence minus normalized binary entropy). The highest-scoring graph $G^* = \arg\max_i \text{Score}(G^{(i)})$ provides the final answer via its terminal node.

Key Contributions¶

Three-agent modular architecture instantiating distinct cognitive inference modes within a single LLM-based pipeline
FRG construction from free-form reasoning traces using NLI-derived edge weights
NLI-calibrated Bayesian belief propagation for coherence scoring without any fine-tuning
$O(k \cdot s)$ end-to-end complexity, more efficient than $O(n)$ sampling-based methods
Demonstrated consistent improvements over CoT, Self-Consistency, and CoT-Decoding on two benchmarks across three LLMs

Results¶

WebOfLies, Mistral-7B: ToTh 70% vs. CoT-Greedy 41% (+29%), Self-Consistency 48%, CoT-Decoding 54%
WebOfLies, DeepSeek-7B: ToTh improves over CoT-Greedy by 14%
WebOfLies, Phi-3.5 Mini: ToTh within 3% of top method (CoT-Decoding 99%, ToTh 96%)
MultiArith, Mistral-7B: ToTh surpasses CoT-Decoding by 4–5 points on Mistral-7B and DeepSeek-7B
Robustness under complexity (Mistral-7B, WebOfLies 5-statement): ToTh 43% vs. CoT-Greedy 19%, Self-Consistency 38%, CoT-Decoding 46%
MultiArith $d_0/l_3$: ToTh 59% (best); $d_0/l_4$: ToTh 45% (best); $d_2/l_3$: ToTh 21% vs. CoT-Decoding 24% (competitive)
Self-Consistency severely underperforms on symbolic tasks: 14% on WebOfLies with DeepSeek-7B

Limitations¶

Fixed three-way cognitive decomposition may misfit tasks requiring hybrid or atypical reasoning patterns (e.g., creative or ambiguous prompts)
Bayesian propagation is sensitive to noise in low-confidence nodes; errors in early steps can cascade through deeper graph regions
Propagation is uniform and unregularized, lacking robustness against adversarial or inconsistent intermediate steps
No dynamic agent routing based on input semantics; all three agents always run regardless of task fit

Relevance to Harnesses / Meta-Harnesses¶

ToTh is a concrete example of a reasoning meta-harness: it orchestrates multiple specialized sub-agents in parallel, aggregates their structured outputs, and applies a formal selection criterion (graph scoring) to arbitrate the final answer—a pattern central to meta-harness design. The NLI-calibrated belief propagation functions as an internal verification layer, analogous to critic or judge agents in broader harness architectures. The $O(k \cdot s)$ linear complexity result is directly useful for harness designers estimating overhead when adding verification stages. Future directions—dynamic agent routing and inter-agent collaboration protocols—point toward adaptive meta-harnesses that compose reasoning modes on demand.