Skip to content

Reasoning-Driven Multimodal LLM for Domain Generalization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

MLLM reasoning improves domain generalization beyond visual feature invariance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RD-MLDG is a domain generalization (DG) framework that harnesses reasoning chains from MLLMs as a complementary signal to feature-level invariance. Two key training challenges — hard-to-optimize reasoning supervision and reasoning-pattern mismatch between teacher (GPT-4o) and student (InternVL3-8B) — are addressed via Multi-Task Cross-Training (MTCT) and Self-Aligned Reasoning Regularization (SARR). The method achieves state-of-the-art results across four standard DomainBed benchmarks.

Problem

Existing DG methods enforce visual feature invariance but neglect higher-level, process-level reasoning that is domain-invariant by construction. Naively fine-tuning MLLMs with GPT-4o-generated reasoning chains fails: (i) reasoning-chain supervision converges slower and underperforms direct label supervision under SFT (−0.93%p accuracy on TerraInc despite +43.28%p zero-shot gain), and (ii) GPT-4o chains are semantically rich but misaligned with the model's own generation style, causing optimization inefficiency.

Method

DomainBed-Reasoning is constructed by prompting GPT-4o (without ground-truth labels) to generate five-stage chains — <SUMMARY>, <CAPTION>, <REASONING>, <REFLECTION>, <CONCLUSION> — for every image in PACS, VLCS, OfficeHome, and TerraInc, with rejection sampling to filter incomplete outputs.

RD-MLDG fine-tunes InternVL3-8B with LoRA adapters on two jointly optimized objectives:

  1. MTCT (Multi-Task Cross-Training): Each training image feeds two prompts simultaneously — a reasoning prompt (from DomainBed-Reasoning) and a no-thinking classification prompt. The combined loss is: $\(\mathcal{L}_{\text{MTCT}} = \frac{1}{B}\sum_{i=1}^{B}\left[-\log p_\theta(y_i|x_i, q_{\text{cls}}) - \frac{1}{T_i}\sum_{t=1}^{T_i}\log p_\theta(r_{i,t}|r_{i,<t}, x_i, q_{\text{reason}})\right]\)$ The direct classification term provides a stable gradient anchor that guides reasoning optimization.

  2. SARR (Self-Aligned Reasoning Regularization): The model generates its own reasoning chains (style-consistent with InternVL3-8B) and uses them as self-labeled supervision via iterative relabeling, preserving semantic richness from GPT-4o chains while reducing the pattern-mismatch gap.

Key Contributions

  • DomainBed-Reasoning: a DG benchmark extension pairing every image in four datasets with structured GPT-4o reasoning chains (ground-truth withheld during generation; rejection-sampled).
  • Empirical analysis identifying two concrete failure modes of direct reasoning-chain SFT: optimization gap (+43.28%p zero-shot gain collapses under SFT) and reasoning-pattern mismatch (GPT-4o chains shift only +1.88%p of tokens vs +29.74%p for self-generated chains).
  • MTCT: dual-pathway training that uses direct label supervision as a stable anchor for reasoning-path optimization.
  • SARR: self-labeling loop that reconciles semantic richness of teacher chains with model-aligned reasoning style.
  • State-of-the-art on all four DomainBed benchmarks, establishing reasoning as a viable process-level invariance signal for DG.

Results

  • Average DomainBed (PACS / VLCS / OfficeHome / TerraInc): RD-MLDG achieves 86.89% avg, vs. the best prior MLLM-based method at 83.46% and best CLIP-prompt method at 83.19%.
  • TerraInc (hardest; strongest domain shift): 70.65% vs. 60.49% for the next MLLM baseline and 63.27% for the best CLIP method — a +10%p+ gain.
  • PACS: 91.73% vs. 90.12% for next MLLM baseline.
  • Zero-shot reasoning (no SFT) yields +43.28%p improvement in ground-truth token probability on TerraInc over no-thinking baseline, confirming that reasoning itself encodes domain-invariant cues.
  • Reasoning-chain SFT alone: −0.93%p vs. direct label SFT, motivating MTCT.

Limitations

  • Relies on GPT-4o (a closed, expensive commercial API) for constructing DomainBed-Reasoning; not trivially scalable to arbitrary new datasets.
  • Evaluation confined to image classification benchmarks (PACS, VLCS, OfficeHome, TerraInc); generalization to other vision-language tasks (VQA, captioning) is not demonstrated.
  • Rejection sampling for chain quality introduces additional data curation overhead.
  • Self-alignment in SARR produces simpler, label-focused chains — potentially sacrificing some contextual richness that GPT-4o chains provide.
  • Base model is InternVL3-8B with LoRA; scalability to larger or different MLLM backbones is not studied.

Relevance to Vision-Language Models

This work directly probes a known weakness of MLLMs — their underperformance on standard classification benchmarks relative to CLIP — and proposes a training recipe that exploits VLM reasoning capability as an explicit inductive bias for distribution shift. It connects to CLIP-based prompt-tuning DG literature (CoOp, MaPLe) but moves from representation-level to process-level invariance, an underexplored axis. The DomainBed-Reasoning dataset and the MTCT/SARR training strategies are immediately relevant to practitioners fine-tuning VLMs for robust deployment, especially where reasoning-chain supervision (chain-of-thought SFT) is being used. The observed trade-off between chain informativeness and optimization tractability is a broadly applicable insight for any VLM fine-tuning pipeline using teacher-generated CoT.