Skip to content

StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

🕒 Published (v1): 2026-07-01 05:34 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; stochastic turn depth for visual instruction tuning addresses training-inference discrepancy

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Visual instruction tuning (VIT) for LVLMs typically packs multiple questions per image into multi-turn (multiT) dialogues, yet evaluation is almost exclusively single-turn (singleT), causing a systematic performance gap. StochasT closes this gap by stochastically varying the conversational history depth during training—without dropping any tokens or adding data. A new Balanced Latin Square evaluation protocol is proposed to measure true context-robust accuracy.

Problem

MultiT VIT groups multiple instruction-answer pairs about the same image into a packed conversation, but standard benchmarks evaluate each question in isolation (singleT). This train/test mismatch causes two failure modes: visual attention decay (attention weights from loss tokens to vision tokens weaken as context grows) and contextual overfitting (models learn to rely on preceding turns that are absent at test time). Existing literature largely ignores this discrepancy, and no unified training recipe or balanced evaluation metric exists to jointly optimize both paradigms.

Method

StochasT (Stochastic Turn Depth) reinterprets a packed \(N\)-turn dialogue \(\mathcal{C} = \{(X_q^{(n)}, X_a^{(n)})\}_{n=1}^N\) as a directed tree by stochastically reassigning each turn's parent node rather than its sequential predecessor.

For turn \(n\), the algorithm traverses backwards through nodes \(k = n-1, n-2, \ldots, 1\). At each step it samples a drop probability \(p_k \sim \text{Beta}(\alpha, \beta)\) and a binary mask \(m_k \sim \text{Bernoulli}(1-p_k)\); the traversal stops at the first \(k\) where \(m_k = 1\), which becomes the parent \(P(n)\). If all preceding nodes are dropped, turn \(n\) connects directly to the root (the visual input \(X_v\)). All \(N\) turns are retained for loss computation—no tokens are discarded—but their causal attention masks reflect the sampled tree structure, diversifying the history \(\tilde{H}^{(n)}\) each turn sees. This is identical in spirit to Stochastic Depth for ResNets but operates on conversational context rather than network layers. Setting \(p_n \to 1\) recovers singleT training; setting \(p_n = 0\) recovers standard multiT.

Balanced Latin Square Turn Permutation (BLS) is a benchmark-agnostic evaluation: for \(N\) turns it constructs an \(\tilde{N} \times \tilde{N}\) permutation matrix (padding with a dummy instruction if \(N\) is odd so \(\tilde{N}\) is always even) such that each turn appears exactly once at every dialogue position and immediately precedes every other turn exactly once. Two metrics are defined: - \(\text{CRA}^{(n)} = \frac{1}{\tilde{N}}\sum_j c_{n,j}\) — average correctness across all \(\tilde{N}\) permutations. - \(\text{CRA+}^{(n)} = \prod_j c_{n,j}\) — 1 only if the model is correct under every permutation (strict context-invariant knowledge).

Key Contributions

  • First systematic analysis of the multiT-training / singleT-evaluation discrepancy in LVLM VIT.
  • StochasT: a compute-neutral, data-neutral training strategy that builds a stochastic dialogue tree, training an implicit ensemble of conversational trajectories without extra tokens, data, or objectives.
  • Balanced Latin Square Turn Permutation evaluation with two new robustness metrics (CRA, CRA+), controlling for both positional bias and first-order carryover effects.
  • Empirical demonstration that StochasT strictly dominates multiT baseline and remains competitive with or superior to the token-expensive singleT baseline across five domain datasets and two backbone models.

Results

  • LLaVA-1.5-7B (avg. over iNat-Plant, PathVQA, CoralVQA, TaiwanVQA):
  • StochasT singleT eval: 68.46 vs. MultiT 61.51 (+9.19% relative gain stated in paper, absolute +6.95 pp), vs. SingleT 67.16.
  • StochasT multiT eval: 70.63 vs. MultiT 68.52 (+3.08% relative), vs. SingleT 68.60.
  • MMDU: StochasT 13.10 vs. MultiT 12.68.
  • Qwen2.5-VL-3B:
  • StochasT singleT eval: 77.28 vs. MultiT 71.20 (+8.54% relative), vs. SingleT 76.71.
  • StochasT multiT eval: 80.16 vs. MultiT 77.39 (+3.58% relative), vs. SingleT 77.75.
  • MMDU: StochasT 59.80 vs. MultiT 57.90.
  • SingleT training requires approximately 2Ă— the token budget to match StochasT's singleT performance due to image duplication across unrolled samples.
  • Additional results on Qwen3-VL-32B reported in supplement (not fully shown in provided text).

Limitations

  • StochasT is explicitly suited for conversations where turns are not strongly interdependent; dialogues with strong cross-turn semantic chaining are incompatible.
  • The BLS evaluation similarly assumes approximate turn independence and should not be applied to contextually dependent conversations.
  • Padding with a dummy instruction to enforce even \(\tilde{N}\) in BLS introduces a minor artifact (dummy turn does not contribute to accuracy but does occupy a permutation slot).
  • LoRA fine-tuning is used throughout; full fine-tuning behavior is not evaluated.
  • Beta distribution hyperparameters \((\alpha, \beta)\) require tuning; robustness across different dialogue length distributions is not fully characterized in the provided text.

Relevance to Vision-Language Models

StochasT directly addresses a structural flaw in the standard VIT pipeline—the mismatch between multiT training and singleT evaluation—that affects virtually every LVLM trained on packed dialogue datasets (LLaVA-style data). The finding that multiT training induces visual attention decay has implications for any work studying visual grounding in LVLMs: longer conversational context systematically dilutes vision-token attention, a failure mode independent of model scale. The BLS evaluation protocol is a practical, benchmark-agnostic tool that the community can adopt to stress-test contextual robustness without bespoke benchmarks. For researchers tracking VLM training recipes and evaluation methodology, StochasT offers a zero-cost plug-in regularizer compatible with auxiliary losses (L2T, Vittle, Ross) and novel architectures.