Collective Intelligence with Foundation Models

🕒 Published (v1): 2026-07-06 19:52 UTC · Source: Arxiv · link

Why this paper was selected

Multi-model coordination framework with judge/solver separation; foundational meta-harness pattern

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A modular multi-agent reasoning framework orchestrates heterogeneous foundation models through a four-stage pipeline: independent solution generation by solver agents, structured critique and revision by a dedicated critic agent, consensus synthesis by an aggregator agent, and a multi-faceted scoring module. A controlled ablation study across four configurations isolates framework structure, redundant sampling, and model heterogeneity as distinct factors. Model heterogeneity emerges as the critical driver, producing a \(2.3\times\) improvement in step-wise reasoning accuracy over homogeneous configurations.

Problem

Individual LLMs suffer from hallucinations, error accumulation in multi-step reasoning, and domain-specific performance variance. Existing multi-agent frameworks do not rigorously disentangle whether observed gains stem from the framework's architectural structure (critique/aggregation mechanisms), redundant sampling from a single model, or genuine model diversity — making it unclear how to design effective multi-agent systems.

Method

The framework is a four-stage sequential pipeline:

1. Solver agents each independently generate \(N=3\) draft solutions, self-evaluate, and return the highest-scoring draft \(D^*\).
2. Critic agent (DeepSeek-R1-0528) performs single-pass structured error analysis on each solver's best draft, checking logical consistency, computational correctness, and answer-reasoning alignment, then outputs a revised solution \(S_i'\).
3. Aggregator agent (GPT-OSS-120B) synthesizes all revised solutions \(\{S_1', \ldots, S_k'\}\) into a consensus \(S_\text{consensus}\), weighting reasoning paths by cross-solver agreement and explicitly flagging uncertainty.
4. Scoring module computes three metrics: (a) semantic similarity via cosine distance in sentence-transformer embedding space; (b) numerical correctness as \(\text{NumericScore} = |N_\text{model} \cap N_\text{ref}| / |N_\text{ref}|\); (c) step-wise accuracy averaging per-step scores across a heuristically decomposed reasoning chain.

Solver models are Meta-Llama-3.3-70B-Instruct, NousResearch Hermes-4-405B, and Qwen3-235B-A22B-Instruct-2507. All agents run at temperature 0.2, max 500 tokens, via an OpenAI-compatible API (Nebius AI).

Ablation configurations: - Individual Baseline: single Llama-3.3-70B, no framework - Case 0 (Homogeneous): one solver + critic + aggregator, all Llama-3.3-70B - Case 1 (Redundant Homogeneous): three Llama-3.3-70B solvers + same-model critic/aggregator - Case N (Heterogeneous): three distinct solvers + DeepSeek-R1 critic + GPT-OSS aggregator

Key Contributions

• Modular, role-specialized multi-agent pipeline separating solve, critique, and aggregate stages with formal pseudocode protocols for each agent.
• Controlled ablation that cleanly isolates framework structure (+0.08 overall score), redundant sampling (+0.01), and model heterogeneity (+0.02 overall, but \(+0.37\) in step-wise accuracy) as independent factors.
• Empirical finding that homogeneous frameworks degrade step-wise reasoning quality below the individual baseline (0.27–0.28 vs. 0.50), achieving correct final answers through error cancellation rather than sound reasoning.
• Multi-dimensional scoring combining semantic, numerical, and step-wise evaluation across an eight-domain scientific benchmark (calculus, physics, chemistry, biology, economics, optimization, statistics, mathematics).

Results

• Overall scores: Individual 0.52 → Case 0 0.60 → Case 1 0.61 → Case N 0.63 (+21% relative over baseline)
• Step-wise accuracy: Individual 0.50 → Case 0 0.28 → Case 1 0.27 → Case N 0.64
• Heterogeneous Case N achieves \(2.3\times\) step accuracy over homogeneous configurations (0.64 vs. 0.27–0.28) and \(1.3\times\) over individual baseline
• Case N outperforms the strongest individual solver (Hermes-4-405B) on both overall and step-wise metrics
• Category-level: Chemistry shows strongest improvement (+0.12 over baseline), Calculus and Biology each +0.06
• Homogeneous configurations with respectable overall scores (0.60–0.61) fail to improve step-wise accuracy, confirming correct answers can be reached via flawed reasoning chains

Limitations

• Benchmark restricted to eight quantitative/scientific domains; generalization to open-ended or language-heavy tasks is untested.
• Only one heterogeneous configuration tested; the specific choice of DeepSeek-R1 as critic and GPT-OSS as aggregator is not ablated against alternatives.
• Critique is single-pass; iterative refinement loops are not explored.
• 500-token output cap may truncate complex multi-step solutions, artificially penalizing harder problems.
• Step decomposition relies on heuristic pattern matching ("Step 1:", numbered lists), which may misalign steps for free-form reasoning.
• Paper text truncates before presenting full category-level and difficulty-level result tables, so some quantitative claims cannot be fully verified from the provided text.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instantiation of a meta-harness: a deterministic orchestration layer that routes inputs through typed agent roles (solver → critic → aggregator) with explicit inter-agent protocols and a centralized scoring module — the structural hallmarks of a harness architecture. The ablation study provides actionable design guidance for harness builders, specifically that replicating a single model across roles can harm reasoning quality, making model diversity a first-class architectural concern rather than an optimization. The finding that a homogeneous harness produces correct outputs via flawed reasoning chains is a critical caution for harnesses deployed in high-stakes or auditable contexts, where step-level fidelity matters as much as final-answer accuracy.