Skip to content

ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025

🕒 Published (v1): 2025-11-20 10:15 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChemLabs is a hierarchical multi-agent system designed to solve IChO 2025 chemistry Olympiad problems, paired with ChemO, a new multimodal benchmark built from those problems. Using Gemini-2.5 Pro with structured visual augmentation, the system achieves 93.6/100, surpassing an estimated human gold-medal threshold (~75 points).

Problem

Chemistry Olympiad problems demand interleaved visual-symbolic reasoning (molecular structures, reaction schemes, spectra) that existing benchmarks (e.g., ChemBench) do not capture. Two structural gaps block automated evaluation: (1) problems requiring visual outputs (e.g., drawing molecules) have no machine-readable answer format; (2) it is impossible to isolate whether model failures stem from weak visual perception vs. weak chemical reasoning.

Method

Benchmark construction (ChemO): 59 sub-problems from IChO 2025 are formalized as \(P_{AER} = \{t, V, a, M, r, \tau, \epsilon\}\). Assessment-Equivalent Reformulation (AER) converts visual-output problems into symbolic equivalents (e.g., SMILES strings, numerical answers). Structured Visual Enhancement (SVE) provides optional tool-generated, expert-verified textual encodings of all visual elements (OCSR for structures, parsed reaction templates, mechanistic step decompositions), enabling a diagnostic comparison: Baseline (raw images) vs. Enhanced (images + structured encodings).

ChemLabs framework: A four-component hierarchical pipeline: 1. Manager Agent — reads the full multi-part IChO question, decomposes it into typed sub-tasks (Structure Construction, Quantitative Calculation, Qualitative Identification, Tabular Enumeration, Mechanistic Reasoning), and dynamically schedules solvers rather than following a static routing table. 2. Perception Lab — invoked only for visual sub-tasks; converts images to structured text (chemical entities, spatial relationships, annotations) before passing to the solver, explicitly decoupling perception from reasoning. 3. Solving Lab — five domain-specific solvers emit structured JSON answers; a solving-introspector performs one refinement pass for consistency and format compliance. 4. Audit Lab — two-stage: chem-auditor checks stoichiometry, oxidation states, mass balance, and intermediate plausibility; general-auditor checks JSON format, reasoning coherence, and numerical accuracy. Failures trigger a diagnostic report back to the introspector for revision, followed by re-audit.

Grading combines rubric-based deductive scoring (with RDKit for structure validation, numerical tolerance checking) and LLM-as-a-Judge for partial-credit cases.

Key Contributions

  • ChemO benchmark: first formalized benchmark from IChO problems, covering 9 problems / 59 sub-problems across organic, inorganic, physical, analytical, and biochemistry domains.
  • AER methodology: principled transformation of visual-output chemistry problems into machine-scorable symbolic formats while preserving assessment equivalence.
  • SVE diagnostic mechanism: decouples visual perception from chemical reasoning to identify the primary bottleneck in MLLMs.
  • ChemLabs: hierarchical multi-agent framework with dynamic task dispatch, specialized domain solvers, a perception–reasoning interface, and a dual-auditor self-correction loop.
  • State-of-the-art result: 93.6/100 with Gemini-2.5 Pro + SVE, above the estimated gold-medal threshold of ~75 points.

Results

  • ChemLabs + SVE + Gemini-2.5 Pro: 93.6/100 (estimated gold-medal threshold ≈ 75.0)
  • SVE alone (without ChemLabs) yields substantial gains over baseline, confirming visual perception as the primary bottleneck
  • Baseline models (raw images, no ChemLabs) fall significantly below gold-medal threshold
  • ChemLabs without SVE outperforms single-agent baselines, with SVE providing additive improvement on top

(Full ablation table not reproduced in the provided text excerpt; specific per-problem and per-model breakdowns are referenced but truncated.)

Limitations

  • ChemO contains only 9 problems (59 sub-problems); statistical conclusions are limited by small benchmark size.
  • Human gold-medal threshold is estimated from 2021 IChO statistics and extrapolated via z-score; direct comparison to 2025 contestants is not possible.
  • SVE relies on expert-verified, tool-generated structured encodings — this preprocessing step is not automated end-to-end and may not scale to arbitrary chemistry datasets.
  • AER reformulation equivalence is validated by domain experts but the reformulation process itself introduces manual effort and potential coverage gaps.
  • The benchmark is drawn from a single year's exam (IChO 2025), limiting diversity and raising questions about generalization.

Relevance to Harnesses / Meta-Harnesses

ChemLabs is a textbook example of a task-specific meta-harness: a Manager Agent dynamically decomposes incoming tasks, routes sub-tasks to specialized solver agents, and orchestrates a multi-stage audit-and-refinement loop — all without hardcoded routing rules. The Audit Lab's error-report-then-re-solve cycle is a concrete implementation of self-correcting agent loops, directly relevant to harness designers building verification and retry scaffolding. The AER + SVE construction pattern — transforming non-machine-readable problem formats into harness-compatible representations as a preprocessing stage — is a generalizable technique for any harness that must interface with multimodal or visually-encoded data. The paper also demonstrates that dynamic, content-aware dispatch (as opposed to static pipeline wiring) meaningfully outperforms fixed workflows on complex, heterogeneous tasks.