Skip to content

MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding

🕒 Published (v1): 2026-07-02 10:13 UTC · Source: Arxiv · link

Why this paper was selected

Graph-aware VLM for molecular images; novel architecture bridging chemistry and multimodal understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MolSight is a graph-aware VLM framework for molecular image understanding that injects chemical-bond topology and symbolic SVG annotations directly into vision tokens via two plug-in modules inserted between the vision encoder and LLM decoder. The key insight is that standard vision encoders capture spatial texture rather than atom/bond adjacency, causing structural misalignment that domain fine-tuning alone cannot fix. MolSight outperforms all generalist and molecular-specialist VLMs on SMILES translation, captioning, descriptor estimation, and bioactivity prediction.

Problem

Standard VLMs fail at molecular image understanding because their vision encoders—pretrained on natural images—encode color, texture, and spatial layout rather than sparse atom symbols and chemical-bond connectivity. Vision tokens preserve image-space locality, not molecular graph adjacency, so ring systems, aromaticity, and substructure identity are systematically lost. Simply adapting a VLM to chemistry data (domain fine-tuning) cannot recover these topological semantics; prior specialist VLMs like ChemMLLM-34B still lag significantly behind task-specific optical chemical structure recognition (OCSR) tools.

Method

MolSight builds on Qwen3-VL and introduces two modules between the vision encoder and LLM decoder, jointly forming a graph topology adaptation layer. Both modules consume SVG text \(T_{\text{svg}}\)—atom symbols, bond types, and 2D positions parsed from the molecular image—as a free structural annotation.

Molecular Topology Module (MTM). Given vision tokens \(V = \{v_1,\ldots,v_K\} \in \mathbb{R}^{K \times d}\), a learnable edge predictor produces a soft adjacency matrix: $\(Z_{ij} = \frac{1}{\sqrt{2d_e}}\bigl(\langle h_i^{\text{src}}, h_j^{\text{dst}}\rangle + \langle h_j^{\text{src}}, h_i^{\text{dst}}\rangle\bigr), \quad \hat{A}_{ij} = \sigma(Z_{ij})\)$ where \(h_i^{\text{src}} = W_s v_i\) and \(h_i^{\text{dst}} = W_d v_i\). \(\hat{A}\) then masks multi-head attention so each token aggregates only from chemically bonded neighbors: $\(\alpha_{ij} = \text{softmax}_j\!\left(\frac{\langle W_q v_i, W_k v_j\rangle}{\sqrt{d_h}} \cdot \hat{A}_{ij} - C(1-\hat{A}_{ij})\right)\)$ Edge logits \(Z\) are supervised by ground-truth bond adjacency \(A^*\) from SVG/RDKit parsing via binary cross-entropy: \(\mathcal{L}_{\text{edge}} = \text{BCEWithLogits}(Z, A^*)\).

Molecular Grounding Module (MGM). Topology-aware tokens \(V^{\text{MTM}}\) are used as queries; SVG text embeddings \(E_{\text{svg}} \in \mathbb{R}^{S \times d}\) serve as keys/values in cross-attention, allowing each vision token to absorb atom types, bond connectivity, and positional cues from the SVG annotation.

Training follows a two-stage curriculum: Substage 1 pretrains MTM alone on 249K image–label pairs with \(\mathcal{L}_{\text{task}} + \lambda\mathcal{L}_{\text{edge}}\); Substage 2 adds MGM with \(\mathcal{L}_{\text{task}}\) only; Stage 2 freezes the topology adaptation layer and fine-tunes the LLM decoder with LoRA per downstream task.

Key Contributions

  • Introduction of SVG text as an explicit, zero-cost structural annotation to guide vision token alignment.
  • MTM: a learnable edge predictor that imposes molecular-graph adjacency as a soft mask in Transformer attention over vision tokens.
  • MGM: vision-to-SVG cross-attention that grounds each vision token in symbolic atom/bond semantics.
  • A two-stage progressive pretraining strategy that stabilizes topology learning before semantic grounding.
  • First unified VLM to surpass task-specific OCSR tools on image-to-SMILES translation while remaining a general-purpose model.

Results

SMILES translation (PubChem, 5k test): - MolSight: Tanimoto Avg Sim 0.97, exact ACC 0.84, validity 99.7% - Best prior VLM (ChemMLLM-34B): Avg Sim 0.87, ACC 0.56, validity 97.2% - OCSR reference (MolScribe): Avg Sim 0.98, ACC 0.66; MolSight's ACC exceeds all OCSR tools despite higher Avg Sim of MolScribe

Molecular captioning on MoleculeQA (5,786 questions, 90.4M trainable params): - MolSight total accuracy: 70.90% vs MolCA-1.3B 64.79% (best prior specialist LLM) and GPT-4 53.47% (10-shot) - Category breakdown: Structure 78.38, Source 73.42, Property 51.16, Application 50.42

Physicochemical descriptor estimation (7 properties, avg): - MolSight avg Pearson 0.984, avg MAE 0.49 vs ChemMLLM-34B 0.959 / 2.43; competitive with task-specific Chemprop (0.974 / 0.41) - MW, TPSA, HBD Pearson ≈ 0.996–0.999

Bioactivity prediction (MolVision/MoleculeNet): - MolSight achieves best results in 5 of 6 tasks (BACE, BBBP, HIV, ClinTox, Tox21, ESOL) over both SMILES-based LLMs and generalist VLMs

Ablation: removing MTM or MGM degrades SMILES translation accuracy; removing both (direct fine-tune of base VLM) degrades further, confirming that topology gains cannot be recovered by downstream fine-tuning alone.

Limitations

  • Requires SVG text as an auxiliary input at both training and inference; SVG must be programmatically derived or available—not always guaranteed for hand-drawn or scanned images.
  • Only evaluates 2D chemical structure diagrams; 3D conformational or reaction scheme images are out of scope.
  • Edge supervision \(\mathcal{L}_{\text{edge}}\) requires ground-truth bond adjacency from RDKit/SVG parsing, tying training data quality to reliable structure parsing pipelines.
  • Table 4 (bioactivity) is partially reported in the text; full quantitative comparison is unavailable from the provided excerpt.
  • Pretraining requires 249K image–label pairs; cost of the topology adaptation pretraining stage is not analyzed.

Relevance to Vision-Language Models

MolSight directly addresses the gap between general-purpose VLM visual representations and structured scientific domains, demonstrating that inserting lightweight topology-aware modules (MTM + MGM) into a frozen VLM backbone can unlock structural reasoning far beyond what domain fine-tuning achieves. The SVG-guided cross-attention strategy is a concrete template for augmenting VLMs with symbolic structural priors in other graph-rich domains (e.g., circuit diagrams, biological pathways). The work also highlights an underappreciated failure mode of patch-based vision encoders: their tokens preserve spatial locality, not relational adjacency, which is the dominant structure in many scientific images. Results showing that MolSight surpasses OCSR specialist tools while remaining a unified VLM set a new ceiling for chemically-grounded visual reasoning and will motivate similar topology-injection approaches in multimodal scientific AI.