Skip to content

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

🕒 Published (v1): 2026-07-04 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

RL over interleaved text-image generation; directly informs training unified multimodal agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BRAID frames interleaved text-image-text (TIT-CoT) reasoning as a unified Markov decision process (MDP), enabling end-to-end reinforcement learning over both text and image generation turns via modality-native policy gradients. Prior work applied RL only to the text branch, leaving image generation outside the optimization loop. On seven spatial reasoning and visual perception benchmarks, BRAID improves over its backbone BAGEL by +5.73 avg. and surpasses GPT-4o at 7B parameters.

Problem

Unified multi-modal models (UMMs) can generate interleaved text-image reasoning traces, but existing RL fine-tuning approaches optimize only the text branch; image generation is either handled by a separate supervised stage (COOPER) or left entirely outside the policy gradient (DeepEyes, UniGRPO). This half-optimization prevents credit assignment from propagating across modality boundaries, leaving the image branch unable to learn reasoning-useful visual intermediates from reward signals alone.

Method

BRAID casts a TIT-CoT trajectory \(\tau = T_1, I_1, T_2, I_2, \ldots, T_K, I_K, T_{\text{ans}}\) as a two-level MDP: an outer turn-level MDP treats each text chunk or intermediate image as a macro-action, and an inner step-level MDP unrolls each macro-action into tokens (autoregressive) or denoising timesteps (flow-matching).

A shared trajectory-level advantage \(\hat{A}(s_k, a_k)\) is computed via GRPO group normalization and channeled into modality-native gradients:

\[J(\theta) = \sum_{k \in K_{\text{txt}}} J_{\text{GRPO}}\!\left(\pi_\theta^{\text{txt}}(T_k|s_k);\,\hat{A}\right) - \lambda \sum_{k \in K_{\text{img}}} \mathcal{L}_{\text{NFT}}\!\left(\pi_\theta^{\text{img}}(I_k|s_k);\,\hat{A}\right)\]

For image turns, the advantage is mapped to a soft reward label \(r_k = \sigma(\hat{A}/\upsilon)\) and injected into the DiffusionNFT loss, which uses implicit predictors \(v_\theta^\pm = (1\mp\beta)v_{\theta_\text{old}} \pm \beta v_\theta\) to steer the flow-matching velocity field without requiring likelihood estimation.

To address long-horizon credit assignment, a VLM judge (GPT-5.2) scores each intermediate image on four criteria — Visual Correctness, Visual Fidelity, Reasoning Utility, and Trustworthiness — producing a per-turn process reward \(r^{\text{vis}}\). Its components are normalized independently across the image group \(I_G\) and merged at the advantage level with equal weight (decoupled advantage estimation), avoiding scale sensitivity across reward dimensions.

Key Contributions

  • Unified MDP formulation for interleaved TIT-CoT that places text tokens and image denoising paths under a single RL objective without separate training stages.
  • Modality-native policy gradient coupling: GRPO clipped surrogate for autoregressive text, DiffusionNFT reward-weighted regression for flow-matching image generation.
  • Vision-thinking process reward: a VLM judge supplying dense per-image-turn feedback decomposed into four complementary criteria with decoupled advantage normalization.
  • Empirical demonstration that joint image-branch optimization is more critical than the process reward, and that BRAID's solution space scales with sampling budget (Maj@\(n\)) while baselines plateau.

Results

  • BRAID (7B) achieves 65.19 avg. across seven benchmarks vs. BAGEL 59.46, SFT 61.99, GPT-4o 62.46, Qwen2.5-VL-7B 62.11.
  • Largest gains over BAGEL: SAT +14.00 (spatial reasoning / mental viewpoint transformation) and V*Bench +10.76 (fine-grained visual discrimination).
  • Ablation — removing image-branch RL (\(-\mathcal{L}_{\text{NFT}}\)): avg. drops from 65.19 to ~62.10 (−3.09), VStar −6.00, CV-Bench 2D −4.32.
  • Ablation — removing vision-thinking reward (\(-r^{\text{vis}}\)): avg. drops to 62.68 (−2.51), VStar −4.48, SAT −2.67.
  • Maj@17 on VStar: BRAID 70.7 vs. SFT 67.5 vs. BAGEL 66.0; on SAT: BRAID 61.3 vs. SFT 56.7 vs. BAGEL 52.3 — confirming expanded solution space, not merely output diversity.
  • Sole regression: CV-Bench 3D −1.24 vs. BAGEL (attributed to SFT distributional gap).

Limitations

  • Evaluated on a single 7B hybrid AR-diffusion backbone (BAGEL); generalization to fully autoregressive UMMs or other scales is not demonstrated.
  • The VLM judge (GPT-5.2) introduces a proprietary API dependency and additional inference cost at training time; its quality directly gates the usefulness of the process reward.
  • CV-Bench 3D shows a consistent regression that is attributed but not resolved.
  • The unified MDP formulation requires simultaneous access to flow-matching denoising internals and autoregressive token probabilities, limiting applicability to architectures with exposed image generation parameters.
  • Training details (number of steps, total compute) are referenced to appendices not included in the provided text.

Relevance to Vision-Language Models

BRAID directly addresses the post-training gap in unified VLMs by showing that RL fine-tuning of the image generation branch — not just the text branch — is essential for multi-modal reasoning quality; this challenges the common assumption that VLM RL need only optimize over the token space. The vision-thinking process reward pattern (VLM-as-judge scoring intermediate visual states) is a generalizable technique applicable wherever VLMs generate intermediate representations. The result that a 7B jointly-optimized UMM surpasses GPT-4o on spatial and visual perception tasks is a strong signal for the VLM community that architecture and post-training strategy matter more than scale alone. This work also clarifies the design space between hybrid AR-diffusion and fully-AR unified models for researchers choosing backbone architectures.