CFPO: Counterfactual Policy Optimization for Multimodal Reasoning¶

🕒 Published (v1): 2026-06-22 11:51 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; counterfactual causal RL for VLM reasoning—novel mechanism beyond standard GRPO/PPO

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CFPO adds a counterfactual regularization term to standard RL policy optimization (GRPO/DAPO) for LVLMs by suppressing high-saliency cross-modal attention regions and maximizing the KL divergence between factual and counterfactual policies. This forces the model to ground reasoning in visual evidence rather than language priors. It achieves 3.17%–6.25% gains over GRPO/DAPO and 1.32%–2.13% over the perception-aware PAPO baseline across ten multimodal benchmarks.

Problem¶

Prevailing RL fine-tuning methods (PPO, GRPO, DAPO) optimize only for final-answer correctness, providing no explicit causal signal linking visual evidence to the generated reasoning chain. This leaves LVLMs vulnerable to three failure modes: (1) Cross-Modal Saliency Deficiency — ignoring image evidence and hallucinating visual entities from language priors; (2) Cross-Modal Saliency Misalignment — attending to perceptually salient but semantically irrelevant regions; (3) Cross-Modal Saliency Inertia — correctly grounding perception but treating visual facts as immutable anchors that override logical "what-if" interventions. Existing counterfactual remedies either incur double inference cost (VCD, M3ID) or require curated supervision (DeFacto, CFVLM), while PAPO's pixel-level masking is too coarse to isolate fine-grained semantic cues.

Method¶

CFPO introduces a Cross-Modal Counterfactual Intervention operating directly on the multi-head attention outputs inside the LLM decoder, without external reward models or SFT.

Attention decomposition. For each attention layer, the full attention matrix $A \in \mathbb{R}^{l \times l}$ is split into high- and low-saliency parts: $$A = A^{H\text{-sal}} + A^{L\text{-sal}}$$ The cross-modal sub-matrix $A^c = A[i,j],\; \forall i \in S_q, j \in S_I$ (text-query-to-image-token attention) is thresholded with a $\mu + 2\sigma$ outlier mask: $$M_\text{cross}[i,j] = \mathbf{1}[A^c[i,j] > \mu + 2\sigma]$$ which is zero-padded to a full mask $M_\text{sal} \in \mathbb{R}^{l \times l}$.

Value intervention. High-saliency visual features are replaced with the layer-wise mean over image tokens $\bar{V} = \frac{1}{|S_I|}\sum_{j \in S_I} V_j$, producing the counterfactual attention output: $$Z_\text{cf} = A \odot (1 - M_\text{sal}) \cdot V + A \odot M_\text{sal} \cdot \bar{V}$$

Counterfactual regularization objective. The causal contribution of visual evidence is quantified as: $$\Delta_\text{cf} = \log \pi_\theta(o \mid Z) - \log \pi_\theta(o \mid \mathrm{do}(Z = Z_\text{cf}))$$ The KL divergence between factual and counterfactual policies is approximated as: $$KL_\text{cf} \approx \exp(\Delta_\text{cf}) - \Delta_\text{cf} - 1$$ The full CFPO objective adds this as a regularizer to GRPO/DAPO: $$J_\text{CFPO}(\theta) = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^G \hat{J}_\text{clip} - \beta KL_\text{ref} + \gamma KL_\text{cf} - \eta \text{Ent}\right]$$ Maximizing $KL_\text{cf}$ penalizes responses whose probability is unaffected by visual suppression (i.e., pure language-prior outputs).

Key Contributions¶

Causal taxonomy of LVLM grounding failures: Saliency Deficiency, Misalignment, and Inertia
Attention-level counterfactual intervention via value substitution with a statistical outlier mask ($\mu + 2\sigma$ threshold), requiring no additional models or labeled data
CFPO objective that plugs into GRPO ($\text{CFPO}_G$) and DAPO ($\text{CFPO}_D$) with a single scalar hyperparameter $\gamma$
Faster training convergence and higher sample efficiency than baselines on ViRL39K

Results¶

Evaluated on Qwen2.5-VL-7B trained on ViRL39K (38,870 QA pairs); average accuracy (%) over 8 rollouts across 10 benchmarks:

$\text{CFPO}_G$ vs GRPO: +3.17% avg (+6.69% We-Math, +5.03% MMk12, +4.73% LogicVista)
$\text{CFPO}_G$ vs PAPO$_G$: +1.32% avg (+3.17% on Math-Centric, +0.57% on Real-World)
$\text{CFPO}_D$ vs DAPO: +6.25% avg (+15.22% We-Math, +12.26% MMk12, +8.96% LogicVista)
$\text{CFPO}_D$ vs PAPO$_D$: +2.13% avg (+5.90% MMk12, +1.86% Math-Centric avg)
Absolute scores (CFPO$_D$, Overall AVG): 58.49% vs DAPO 55.60% vs PAPO$_D$ 57.24%
RealWorld-centric (C-VQA-Real): CFPO$_D$ 67.40%, DAPO 65.27%, PAPO$_D$ 65.13%

Limitations¶

Evaluated only on Qwen2.5-VL-3B and 7B; scalability to larger models (e.g., 72B) not demonstrated
The $\mu + 2\sigma$ threshold for saliency masking is a fixed statistical heuristic; robustness across diverse attention distributions not fully characterized
Value intervention replaces high-saliency regions with the mean of image tokens per layer — a coarse non-informative prior that may suppress valid contextual redundancy
Training on ViRL39K only; generalization to other RL training corpora not tested
Causal intervention applied at the attention output level relies on structural abstraction assumptions that may not hold uniformly across architectures

Relevance to Vision-Language Models¶

CFPO directly targets the core alignment failure in LVLM RL fine-tuning: the inability of outcome-only reward signals to enforce visual grounding in chain-of-thought reasoning. It advances the line of work (VCD, PAPO) seeking to move counterfactual grounding from inference-time patching into the training loop itself, and does so without external supervision, making it directly applicable on top of existing GRPO/DAPO pipelines. The $\mu + 2\sigma$ attention-mask mechanism is architecturally agnostic to the visual encoder and language backbone, positioning CFPO as a broadly applicable regularizer for any decoder-only LVLM trained with RL. For VLM researchers, it provides both a causal framing of hallucination failure modes and an empirically validated, low-overhead training recipe.