Skip to content

Do Multi-Agents Solve Better Than Single? Evaluating Agentic Frameworks for Diagram-Grounded Geometry Problem Solving and Reasoning

🕒 Published (v1): 2025-12-18 16:00 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically benchmarks single-agent versus two-stage multi-agent (Interpreter + Solver) pipelines for diagram-grounded geometry problem solving across four visual math benchmarks. Multi-agent decomposition consistently improves open-source models (e.g., Qwen-2.5-VL 7B gains +6.8% on Geometry3K, +9.4% on OlympiadBench) but rarely helps—and sometimes hurts—strong closed-source models like Gemini-2.0-Flash. The key finding is that agentic decomposition is conditionally beneficial, not universally optimal.

Problem

Existing geometry solvers either require fine-tuning, depend on costly manual formalization (AutoGPS), or rely on single-model end-to-end inference whose benefits from agentic decomposition remain unevaluated. No systematic zero-shot comparison of single-agent vs. multi-agent architectures existed across diverse geometry benchmarks spanning difficulty levels from high-school to Olympiad.

Method

The single-agent baseline feeds (image, question, zero-shot prompt) directly to a multimodal LLM to produce an answer. The multi-agent pipeline splits the task into two frozen agents: 1. Interpreter Agent (VLM): receives the diagram and question, outputs a set of formal geometric predicates/literals (e.g., Parallel(Line(D), Line(H)), MeasureOf(Angle(D,A,B), 38)) via a structured prompting protocol. 2. Solver Agent (LLM): receives the predicates + original question and performs symbolic reasoning to produce the final answer.

The Interpreter prompt enforces a six-step chain-of-thought (shape identification → circle analysis → angle analysis → congruence → intersection → constraint synthesis) before emitting predicates. Semantic fidelity of predicates is measured via cosine similarity of embeddings (OpenAI text-embedding-3-large) between descriptions derived from the original image versus from the generated predicates. Free-form answer evaluation uses Gemini-2.0-Flash as an LLM judge. Performance is reported using Pass@3 on three independent runs.

Key Contributions

  • Systematic zero-shot comparison of single-agent vs. two-stage multi-agent pipelines across four benchmarks: Geometry3K (601 MC), MathVerse (788 problems), OlympiadBench (8,476 problems), We-Math (6,500 problems).
  • Empirical demonstration that multi-agent benefit is model-capacity-dependent: consistent for open-source models, marginal or negative for strong proprietary models on classic benchmarks.
  • Ablation showing Interpreter quality is the binding constraint: Qwen-2.5-7B interpreter yields 35.77% (Phi-4 solver) vs. 70.05% with Gemini interpreter—a 34-point gap.
  • Predicate alignment analysis via embedding cosine similarity as a proxy metric for symbolic-to-visual fidelity (Gemini predicates: 0.849, GPT-4o mini: 0.794, Qwen-2.5: 0.677).
  • New zero-shot SOTA on Geometry3K (83.19%) and MathVerse (69.67%) using Interpreter–Solver with significantly fewer parameters than prior methods.

Results

  • Geometry3K: Qwen-2.5-VL 7B: 60.07% (multi) vs. 53.24% (single) +6.8pp; Qwen-2.5-VL 32B: 72.05% vs. 68.72% +3.3pp; Gemini-2.0-Flash: 83.86% vs. 85.19% −1.3pp.
  • MathVerse: Qwen-2.5-VL 7B: 46.19% vs. 52.16% −6.0pp; Qwen-2.5-VL 32B: 67.77% vs. 66.67% +1.1pp; Gemini: 74.68% vs. 75.13% −0.45pp.
  • OlympiadBench: Qwen 7B: +9.4pp (61.84% vs. 52.44%); Qwen 32B: +6.67pp (64.56% vs. 57.89%); Gemini: −2.46pp.
  • We-Math: All three models improve with multi-agent (+2.66pp, +4.64pp, +1.74pp), including Gemini—the only benchmark where the proprietary model benefits.
  • vs. baselines: Interpreter–Solver–Gemini reaches 83.19% on Geometry3K (vs. AutoGPS 81.6% at ≈200B params); Interpreter–Solver–Qwen3 reaches 69.67% on MathVerse (vs. OpenVLThinker 47.9%).

Limitations

  • All experiments are zero-shot only with frozen parameters; no fine-tuning ablation, so it is unknown whether decomposition advantages persist under supervised training.
  • Fixed prompting setup—no adaptive, iterative, or self-refining prompting strategies tested.
  • Open-source models run at 4-bit quantization (Unsloth), which may suppress full-precision performance.
  • Only two model families (Qwen-2.5-VL variants, Gemini-2.0-Flash/GPT-4o mini) evaluated; no insight into behavior at larger open-source scales (e.g., 70B+).
  • Predicate quality not directly measured against gold annotations (no gold predicate labels in any benchmark); cosine similarity of re-derived descriptions is a weak proxy.
  • No systematic search over alternative literal extraction strategies or predicate schemas.

Relevance to Harnesses / Meta-Harnesses

This paper is directly relevant as an empirical study of how task decomposition into specialized sub-agents—a core pattern in agentic harnesses—affects end-to-end accuracy. It provides concrete evidence that the Interpreter→Solver pipeline structure (a two-stage harness) is conditionally beneficial, with the upstream component's output quality being the binding constraint on the entire pipeline—a key architectural insight for harness designers. The ablation result (34-point accuracy swing from Interpreter quality alone) quantifies the cost of a weak upstream agent in a chained harness, directly informing decisions about which components to allocate compute/model capacity to. The finding that single-agent mode can outperform decomposition for strong models also raises the meta-question of when to dynamically route between harness architectures, pointing toward adaptive meta-harness designs.