MuSLR: Multimodal Symbolic Logical Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MuSLR introduces the first benchmark and task definition for multimodal symbolic logical reasoning, requiring VLMs to apply formal logic rules (modus ponens, modus tollens, etc.) over joint image-text inputs. Evaluation of 7 state-of-the-art VLMs shows even the best (GPT-4.1) reaches only 46.84%. The authors also propose LogiCAM, a modular CoT framework that boosts GPT-4.1 by 14.13% by decomposing reasoning into premise selection, reasoning-type identification, and iterative symbolic deduction.

Problem

No benchmark or formal task definition existed for symbolic logical reasoning grounded in multimodal (image + text) inputs. Prior symbolic reasoning benchmarks (FOLIO, ProofWriter, Multi-LogiEval) are text-only; prior multimodal benchmarks (LogicVista, MMMU, MathVista) do not require explicit application of named formal inference rules. Neuro-symbolic pipelines that use theorem provers require converting visual content to text first, incurring information loss.

Method

MuSLR-Bench construction: Images collected from COCO, Flickr30k, nocaps, MIMIC, RVL-CDIP, ScienceQA, and traffic reports. GPT-4o extracts fine-grained visual details. Symbolic rules from propositional logic (PL), first-order logic (FOL), and non-monotonic logic (NM) are combined into abstract reasoning chains (depth 2–9), then grounded in real-world text retrieved from healthcare, traffic, and Wikipedia. Questions (Truth Evaluation: True/False/Unknown; Multiple Choice: 4-way) are generated via rule-based substitution. Automatic (lexical similarity, commonsense plausibility) and manual quality control applied.

LogiCAM framework (GPT-4.1 backbone, 3-shot CoT): 1. Premise Selector: identifies the symbolic rules relevant to the question, then extracts the image regions/details pertinent to those rules, producing I_critical. 2. Reasoning Type Identifier: decides whether formal symbolic reasoning or heuristic commonsense applies at the current step. 3. Reasoner: applies the selected inference rule (syllogism form) or commonsense to derive a conclusion C. 4. Completion Check: if C does not yet answer the question, appends C to context and iterates.

Key Contributions

• First formal definition and benchmark for multimodal symbolic logical reasoning (MuSLR task + MuSLR-Bench).
• MuSLR-Bench: 1,093 instances across 7 domains, 3 logic types (PL/FOL/NM), 35 atomic rules, 976 rule combinations, depths 2–9.
• LogiCAM: modular CoT baseline achieving state-of-the-art on MuSLR-Bench (+14.13% over GPT-4.1 CoT).
• Comprehensive error and depth analysis identifying cross-modal logical misalignment (~67–74% of failures) as the dominant failure mode.

Results

• GPT-4.1 (best baseline): 46.84% overall; InternVL3-8B best open-weight at 45.20%.
• LogiCAM (GPT-4.1 + framework): +14.13% overall over GPT-4.1 CoT.
• Logic-type gains for LogiCAM over GPT-4.1: FOL +48.93%, PL +31.93%, NM +26.17%.
• Depth robustness: LogiCAM 71.91% at depth 2–3, 54.61% at depth 8–9; surpasses GPT-4.1 by 13% at deepest levels.
• Reasoning traceability (LogiCAM): ROUGE-L 0.170, BertScore-F1 0.835, ROSCOE mean 0.590 — highest across all models.
• Accuracy by logic type (all models avg): NM 46.09% > PL 42.77% > FOL 37.04%.
• Ablation: removing symbolic reasoning module costs −5.14%, removing heuristics −3.45%, removing premise selection −3.27%.

Limitations

• LogiCAM is built solely on GPT-4.1; generalizability to other VLM backbones is not directly demonstrated.
• Despite improvements, LogiCAM still declines with depth (71.91% → 54.61%), indicating unsolved long-chain reasoning robustness.
• ~67% of LogiCAM's own errors are still cross-modal logical misalignment, the same dominant failure mode as baselines.
• Benchmark size is modest (1,093 instances); domain coverage, while diverse, relies on GPT-4o-generated visual descriptions, which may introduce its own biases.
• FOL instances require precise quantifier binding and variable tracking that VLM-based approximation (LogiCAM) handles better but still imperfectly.

Relevance to Vision-Language Models

MuSLR directly stress-tests a capability gap in VLMs that current multimodal benchmarks do not isolate: the ability to fuse visual and textual evidence under formal, verifiable logical rules rather than heuristic pattern matching. The finding that ~70% of errors stem from cross-modal logical misalignment — not perception or language understanding in isolation — points to a structural weakness in how current VLMs ground multimodal evidence into coherent symbolic representations. LogiCAM's modular decomposition (premise selection → type identification → iterative deduction) offers a design template for neuro-symbolic augmentation of VLMs that avoids the lossy text-conversion step required by classic theorem provers. For researchers tracking VLMs, MuSLR provides a principled, difficulty-scaled diagnostic to measure progress on rigorous, multi-hop multimodal reasoning.