Revealing Multimodal Causality with Large Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MLLM-CD is a framework for causal discovery from multimodal unstructured data (text + images) that uses MLLMs to identify causal variables and refine causal graph structures. It combines contrastive factor discovery, statistical structure learning (FCI/PC), and iterative counterfactual reasoning to overcome two core failure modes of prior LLM-based causal discovery: incomplete variable identification and structural ambiguity. Evaluated on GPT-4o, Gemini 2.0, LLaMA 4 Maverick, and Grok-2v, it consistently outperforms baselines in both factor and structure recovery.

Problem

Existing LLM-driven causal discovery (e.g., COAT) operates on unimodal text and fails in multimodal settings for two reasons: (1) it misses implicit causal variables that only become identifiable through intra- or inter-modal interactions (e.g., linking lesion size in CT images to clinical note findings), and (2) statistical causal discovery from purely observational data leaves many edges undirected due to Markov equivalence, and this ambiguity is amplified when the variable count grows with modality count.

Method

MLLM-CD chains three modules:

1. Contrastive Factor Discovery (CFD): CLIP-based embeddings are used to select maximally-distant sample pairs within each modality (intra-modal) and maximally-misaligned pairs across modalities (inter-modal, scored by semantic distance plus target-variable difference). These contrastive pairs are presented to an MLLM with dedicated prompts to surface implicit causal variables. Candidate factors are merged and deduplicated, then the MLLM annotates factor values for each sample, converting unstructured data into a structured dataset.

2. Causal Structure Discovery: A standard statistical algorithm (FCI by default, which handles latent confounders) is run on the structured dataset to infer a partial ancestral graph.

3. Iterative Multimodal Counterfactual Reasoning (MCR): For uncertain/undirected edges in the graph, the MLLM performs "what-if" interventions — predicting counterfactual factor values and generating new multimodal samples (revised text + image-generation model for visuals). Generated counterfactuals pass two validation checks: semantic plausibility (cosine similarity ≥ τ_sem) and causal consistency (non-descendants of the intervened variable must not change beyond threshold τ_causal). Validated samples augment the dataset and the CFD→structure→MCR loop repeats.

Key Contributions

• First framework for causal discovery from multimodal unstructured data using MLLMs, extending prior LLM-CD work (COAT) to the multimodal setting.
• Contrastive factor discovery module exploiting both intra- and inter-modal contrastive pairs to surface implicit causal variables.
• Iterative multimodal counterfactual reasoning module that generates and validates multimodal counterfactual samples to resolve structural ambiguities beyond observational data.
• First benchmark for multimodal unstructured causal discovery: synthetic MAG dataset (200 samples, 9 factors, 3 visual + 5 verbal) and real-world Lung Cancer dataset (60 samples, 5 factors, image + clinical text).
• Extended Structural Hamming Distance (ESHD) metric that jointly penalizes factor mismatches and edge errors.

Results

• MAG dataset (averaged over GPT-4o, Gemini 2.0, LLaMA 4, Grok-2v): MLLM-CD achieves NF=0.89 vs. best baseline META at 0.52; AF=0.57 vs. COAT at 0.29; ESHD=13.42 vs. COAT at 16.42 (lower is better).
• Lung Cancer dataset (averaged): MLLM-CD achieves NF=0.90 vs. META at 0.45; AF=0.68 vs. META at 0.59; ESHD=5.33 vs. COAT at 9.42.
• Ablation (Gemini 2.0, MAG): Removing CFD drops NF from 0.87 to 0.73 and AF from 0.60 to 0.47; removing MCR drops ESHD from 14.00 to 15.67.
• CFD sampling ablation (MAG): Combined intra+inter-modal strategy (NF=0.87) outperforms intra-only (NF=0.82), inter-only (NF=0.63), and random (NF=0.73).
• MLLM-CD achieves best ESHD across all four tested LLMs on both datasets.

Limitations

• Benchmark datasets are small (200 and 60 samples); ground-truth causal graphs rely on domain expert annotation, limiting scalability evaluation.
• Modality coverage is bounded by the underlying MLLM — specialized modalities (sensor signals, genomic sequences) are not supported.
• MLLM hallucinations and training-data biases can corrupt counterfactual generation despite validation filters; the causal consistency check itself uses an imperfect intermediate graph.
• No rigorous identifiability guarantees; effectiveness is empirically driven.

Relevance to Vision-Language Models

MLLM-CD is a direct application of MLLMs as causal reasoning engines over paired image-text data, stress-testing their ability to detect semantically subtle cross-modal relationships — a capability central to VLM research. The contrastive prompting strategy (exploiting CLIP embeddings to select maximally informative pairs) offers a reusable pattern for eliciting fine-grained visual-verbal reasoning from MLLMs beyond standard VQA. The counterfactual generation pipeline, which requires the MLLM to produce causally coherent image descriptions alongside modified text, probes compositional and interventional understanding that current VLM benchmarks rarely target. For VLM researchers, this work introduces causal discovery as a new evaluation axis and a methodology for using contrastive multimodal samples to surface what MLLMs do and do not understand about cause-and-effect in visual scenes.