SportD: Can VLMs Physically Strategize?¶

🕒 Published (v1): 2026-07-16 06:30 UTC · Source: Arxiv · link

Why this paper was selected

Probes VLM physical strategy/reasoning in soccer; novel eval axis for agent decision-making

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SportD is a benchmark of 478 on-ball soccer decisions from the 2022 FIFA World Cup that evaluates whether frontier VLMs can make strategically optimal decisions—shoot or pass to a specific teammate—scored against a possession-value model (VAEP). All three tested VLMs fall below professional players on accuracy, skill, and regret, while exhibiting systematic risk-aversion and partial imitation of human play patterns rather than true counterfactual evaluation.

Problem¶

Existing VLM benchmarks test perception, captioning, and question answering—representations of scenes rather than decisions within them. No prior benchmark grades VLM forward-looking, on-ball action choices against an objective value function, leaving the question of physical strategic reasoning unaddressed. The gap between strong visual understanding and strategically effective action selection is unmeasured.

Method¶

Each SportD item presents a VLM with three aligned views of a World Cup decision moment: (i) ~10 images sampled at 0.5 s intervals over the preceding ~5 s, (ii) a freeze-frame at decision time, and (iii) a top-down map with lettered (identity-withheld) player markers. The VLM must select one action from \(A_e = \{\text{SHOOT}\} \cup \{\text{pass to teammate } \ell : \ell \in T_e\}\) (median ~6 options).

Actions are valued with VAEP: the state value is \(V = P_{\text{score}} - P_{\text{concede}}\), estimated by gradient-boosted trees (XGBoost, 100 trees, depth 3) trained on Euro 2020/2024 and applied to the 2022 World Cup. Expected action value is \(V(a) = p_a v_a^+ + (1-p_a) v_a^-\). The main metrics are: - Regret: \(R = V^\star - V \geq 0\) (value forfeited vs. optimal action \(a^\star\)) - Skill: \(s = (V(\hat{a}) - \bar{V}) / (V^\star - \bar{V})\), scale-free, 1 = optimal, 0 = random - Imitation rate: \(\Pr[\hat{a} = a_h \mid a_h \neq a^\star, \hat{a} \neq a^\star]\) vs. a chance baseline - Risk: per-action outcome variance \(\text{Var}(a) = p_a(1-p_a)(v_a^+ - v_a^-)^2\), compared via Wilcoxon signed-rank

Events are stratified into low/medium/high \(\Delta\) terciles based on how suboptimal the player's own choice was.

Key Contributions¶

SportD benchmark: 478 value-grounded, on-ball soccer decision instances from the 2022 FIFA World Cup with quantitative regret scoring.
Possession-value evaluation pipeline: adaptation of VAEP to benchmark VLM action choice, scoring both exact-match accuracy and the magnitude of value forfeited.
Risk-aversion finding: systematic quantification that frontier VLMs choose lower-variance, lower-reward actions than both the optimum and real players on 63–72% of events (\(p < 10^{-40}\)).
Imitation finding: VLMs reproduce the human player's exact (often suboptimal) action well above the ~22% chance rate (GPT: 44%, Gemini: 31%, Claude: 32%), distinguishing pattern imitation from counterfactual evaluation.
∆-stratified analysis: design that cleanly separates imitative from evaluative behavior by checking whether human-agreement falls off as the human's choice becomes increasingly suboptimal.

Results¶

Optimal-action accuracy: Gemini 3.5 Flash 31.4%, GPT 5.6 Sol 31.0%, Claude Opus 4.8 23.2% vs. real players 38.9% and random baseline 19.3%.
Skill: Gemini/GPT 0.239, Claude 0.142 vs. players 0.320 and random 0.007.
Mean regret: Gemini +0.0351, GPT +0.0337, Claude +0.0457 vs. players +0.0275 (goal-probability units); all VLMs significantly above players.
Top-20% accuracy: VLMs 35.6–42.9% vs. players 52.5%; real players lead at every rank-percentile threshold.
Risk-aversion: every VLM picks lower-variance actions than \(a^\star\) on 63–72% of events; mean chosen action value 0.049–0.061 vs. 0.068 (players) and 0.096 (optimum). Shot rate: VLMs 12.1–25.7% vs. players 33.7%.
Pass progression: optimal pass advances ball +8.8 m goal-ward; VLMs: Gemini +1.7 m, Claude −0.2 m, GPT −1.6 m; VLMs go forward on only 40–61% of passes vs. 56% for real players.
Imitation rate: GPT 44% (\(p < 10^{-15}\)), Gemini 31% (\(p = 4\times10^{-4}\)), Claude 32% (\(p = 2\times10^{-4}\)); all significantly above chance ~22%.
∆-stratified imitation: on high-\(\Delta\) events, only GPT sustains the human-lean (33.6% vs. 19.1%); Gemini and Claude converge to chance.

Limitations¶

Coverage restricted to a single competition (men's 2022 FIFA World Cup) due to data availability constraints.
Action-value model uses relatively simple gradient-boosted trees (XGBoost), providing an approximation rather than ground-truth strategic value; no uncertainty estimates or robustness analyses across alternative valuation methods are provided.
Action space is limited to passing and shooting; carrying, dribbling, holding, and off-ball movement lie in continuous spaces and are excluded.
Only three frontier VLMs tested; results may not generalize across all model families.
The benchmark is small (478 events), making fine-grained subgroup analyses (e.g., by game state or match context) statistically limited.

Relevance to Vision-Language Models¶

SportD directly probes a capability gap in frontier VLMs—physical strategic reasoning—that standard perception and QA benchmarks cannot detect: a model can ace scene description while still making systematically suboptimal action choices. The finding that VLMs exhibit risk-aversion and pattern imitation rather than counterfactual value evaluation has implications for VLM deployment in any agentic, embodied, or decision-making role, complementing work on VS-Bench and SPORTU by grounding evaluation in real-world continuous action consequences. For researchers tracking VLMs, the VAEP-based regret framing offers a reusable template for converting any domain with an objective payoff model into a value-scored VLM decision benchmark.