Skip to content

Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature

🕒 Published (v1): 2026-06-29 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MatMMExtract is an end-to-end open-source pipeline that decomposes compound scientific figures from materials science literature into sub-panels and annotates them with LLM-generated structured metadata, producing MatSciFig—391,606 panel-level image-text pairs from 180,571 figures across 14,810 articles. A domain-specific compound-figure detector (MaterialScope, 2,811 annotated figures) achieves mAP50 of 0.9227 with a fine-tuned YOLO12-m. A CLIP-style dual-encoder trained on MatSciFig yields a 4.4× improvement in R@1 over zero-shot CLIP, confirming dataset utility for vision-language pretraining.

Problem

General-purpose VLMs (CLIP, BLIP, LLaVA) fail on domain-specific scientific imagery because no large-scale, modality-diverse, panel-level image-text dataset exists for materials science. The core structural obstacle is that ~62% of materials science figures are compound—a single caption describes multiple sub-panels simultaneously—making direct image-caption pairing noisy and semantically underspecified. Existing resources (Exsclaim, MatCha, MatQnA, MATRIX, MatMech) are either narrow in modality scope, limited to evaluation benchmarks, or focused on causal reasoning rather than diverse image-text training pairs.

Method

The pipeline has three sequential stages:

  1. Data collection: Articles (CC-BY, CC-BY-NC) are retrieved from Elsevier and Springer via publisher XML APIs after metadata filtering through OpenAlex and Scopus. Each figure yields a triplet \(\langle\text{image},\,\text{caption},\,\text{in-text reference sentence}\rangle\).

  2. Compound figure detection: A domain-specific detector localises sub-panel bounding boxes. To close the biomedical-to-materials domain gap, the authors manually annotated 2,811 materials science figures (12,906 boxes) into the MaterialScope benchmark, then fine-tuned six detector architectures (YOLO8/9/10/11/12-m, DAB-DETR) on it. YOLO12-m was selected as the pipeline detector.

  3. LLM annotation (text-only): For each localised sub-panel, an LLM receives only the textual caption and in-text reference—not the panel image—and produces: (i) a grounded sub-caption, (ii) a visualisation category + subtype from a curated two-level taxonomy (19 categories, 100+ subtypes), and (iii) a 40–60-word scientific summary. Output is constrained via JSON schema (strict enumeration for category) and generated at temperature 0.1 with chain-of-thought disabled for throughput.

Downstream baseline: A CLIP ViT-B/32 visual encoder paired with MatSciBERT text encoder is trained on MatSciFig and indexed with FAISS for retrieval evaluation.

Key Contributions

  • MatMMExtract pipeline: Open-source, license-aware pipeline released as a PyPI package (matmmextract) that produces panel-level \(\langle\text{image},\,\text{sub-caption},\,\text{category},\,\text{summary}\rangle\) records.
  • MaterialScope benchmark: 2,811 manually annotated materials science figures (12,906 boxes, inter-annotator \(\kappa = 0.9245\)) for compound figure detection, with detector benchmarks across six architectures.
  • MatSciFig dataset: 391,606 annotated panels from 180,571 figures—the largest panel-level vision-language dataset for materials science.
  • Materials science taxonomy: Two-level, 19-category Ă— 100+ subtype classification schema covering microscopy, diffraction, spectroscopy, mechanical testing, electrochemistry, simulation, etc.
  • LLM annotation benchmark: Six low-cost LLMs evaluated for category classification accuracy and sub-caption/summary quality (including hallucination rate) on 350 compound figures.
  • Retrieval baseline: Dual-encoder achieving 4.4Ă— R@1 improvement over zero-shot CLIP on MatSciFig.

Results

  • Compound figure detection (MaterialScope test set, 281 images):
  • YOLO12-m: mAP50 = 0.9227, F1 = 0.9502, 13.76 img/s
  • YOLO8-m: mAP50 = 0.9028 (fastest at 14.31 img/s)
  • DAB-DETR: mAP50 = 0.6569 (slowest at 9.18 img/s, highest parameters at 43.65M)
  • Exsclaim (prior SOTA, limited to A–H panels at 640Ă—640): mAP50 = 0.7902; YOLO12-m exceeds it by +13.3 pp
  • Category classification (350 compound figures):
  • Gemini 3.5 Flash strongest overall: 99.0% on Microscopy, 97.8% on Mechanical Test
  • Gemini 3.1 Flash Lite close behind; selected for pipeline (best cost-quality trade-off)
  • GPT-5.4 Nano weakest: 43.2% on Photograph, 55.6% on Simulation
  • Sub-caption/summary quality:
  • Gemini 3.1 Flash Lite: 82% of outputs rated good, hallucination rate 4.8% → selected as pipeline annotator
  • Retrieval:
  • Dual-encoder on MatSciFig: 4.4Ă— improvement in R@1 over zero-shot CLIP

Limitations

  • LLM annotation is text-only (caption + reference sentence); the panel image is never seen by the LLM, so annotations depend entirely on caption quality and completeness—visually distinctive content not described in text (e.g., "Photograph" category) is poorly classified.
  • Dataset is restricted to Elsevier and Springer open-access articles with CC-BY/CC-BY-NC licenses; coverage skews toward journals on these platforms.
  • Hallucination rate of 4.8% (best model) means a non-trivial fraction of annotations contain fabricated content; downstream models trained on MatSciFig inherit this noise.
  • Sub-panel label distribution is heavily skewed (labels A–E dominate; T has only 3 examples), limiting detector performance on rare panel configurations.
  • The downstream retrieval baseline uses only CLIP ViT-B/32 + MatSciBERT; no fine-tuned generative VLM is evaluated on MatSciFig.
  • Paper text is truncated; sub-caption/summary quality evaluation (beyond hallucination rate) is not fully reported in the provided excerpt.

Relevance to Vision-Language Models

MatSciFig directly addresses the data bottleneck for domain-adapted VLMs in materials science, providing the scale and panel-level grounding that zero-shot CLIP-family models lack—the 4.4× R@1 gain from a simple dual-encoder illustrates how much headroom domain-specific data unlocks. The structured taxonomy and hallucination-audited LLM annotation pipeline offer a replicable blueprint for bootstrapping domain VLM training data from compound-figure-heavy scientific literature, applicable beyond materials science. The MaterialScope detector benchmark also fills a methodological gap: reliable sub-panel localisation is a prerequisite for any pipeline that wants to move from figure-level to panel-level vision-language alignment in scientific domains. Researchers building or fine-tuning VLMs (e.g., LLaVA-style models) for scientific imagery will find both the dataset and the pipeline directly actionable.