Skip to content

HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

🕒 Published (v1): 2026-03-27 12:42 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; benchmark diagnosing fine-grained hand spatial reasoning failures in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HandVQA is a large-scale diagnostic benchmark of 1.6M+ multiple-choice questions probing VLMs' ability to reason about 3D spatial relationships between hand joints (angles, distances, relative positions). Base VLMs perform at or below random chance on most subtasks; LoRA fine-tuning substantially recovers performance and yields zero-shot transfer gains of +10.33% on gesture recognition and +2.63% on hand-object interaction recognition.

Problem

VLMs achieve near-human accuracy on general VQA benchmarks yet lack genuine geometric grounding for fine-grained spatial tasks. No prior benchmark targeted the 21-joint kinematic structure of human hands—where errors in joint-angle or distance interpretation cause cascading failures in robotics, surgical assistance, and AR/VR gesture interfaces. Existing spatial benchmarks (SPHERE, What'sUp) test inter-object relationships rather than intra-object, part-whole geometry.

Method

Data generation pipeline operates on normalized 3D hand joint coordinates \(P = \{\mathbf{p}_i \in \mathbb{R}^3\}_{i=1}^{21}\) from FreiHAND, InterHand2.6M, and FPHA:

  1. Pose descriptor extraction (\(F_{\text{pose}}\)): Computes continuous descriptors \(\Psi\) comprising joint-bending angles: $\(\theta_j = \arccos\frac{(\mathbf{p}_{a(j)}-\mathbf{p}_j)\cdot(\mathbf{p}_{b(j)}-\mathbf{p}_j)}{\|\mathbf{p}_{a(j)}-\mathbf{p}_j\|\,\|\mathbf{p}_{b(j)}-\mathbf{p}_j\|}\)$ Euclidean pairwise distances \(d_{(i,k)} = \|\mathbf{p}_i - \mathbf{p}_k\|_2\), and signed axis offsets \(\Delta_a(i,k) = \langle \mathbf{p}_i - \mathbf{p}_k, \mathbf{a}\rangle\). Each is discretized via fixed thresholds into ordinal category labels (e.g., angle bins: bent completely inward \(\theta < 105°\), bent inward, bent slightly inward, straight \(\theta \geq 170°\); distance bins: close/spread/spread wide; relative-position bins: left/aligned/right per axis). Aligned cases are excluded.

  2. Sentence generation (\(F_{\text{text}}\)): Fills deterministic syntactic templates with joint names and category labels to produce candidate answer sentences; the true category label becomes the correct option, others serve as distractors.

  3. MCQ formation (\(F_{\text{mcq}}\)): Randomly samples 5 joint/joint-pair instances per descriptor type, yielding 25 MCQs per image across 5 subtasks. Scales to 107 possible MCQs per image before sampling.

Evaluation: LLaVA Mistral 7B, DeepSeek Janus Pro 7B, and Qwen 2.5 VL 7B Instruct are assessed in base and LoRA-finetuned settings. Ordinal subtasks (angle, distance) use both accuracy and MAE; binary relative-position subtasks (X/Y/Z) use accuracy only.

Key Contributions

  • A fully automated, 3D-grounded pipeline generating >1.6M controlled MCQs spanning 5 spatial subtasks over hand joint geometry.
  • Systematic evaluation of three leading 7B VLMs revealing base models at or below random chance on distance and angle tasks, with characteristic failure modes (mode collapse toward "close" or "slightly bent" predictions).
  • Demonstration that LoRA fine-tuning on HandVQA instills transferable 3D spatial knowledge, enabling zero-shot gains on gesture recognition (+10.33%) and video-based hand-object interaction (+2.63%) without task-specific training.

Results

Base models (no fine-tuning): - Angle accuracy: LLaVA 23.4–42.5%, Qwen 24.2–38.7%, DeepSeek 26.5–35.3% (random baseline = 25%) - Distance accuracy: LLaVA 13.2–16.2%, Qwen 18.0–20.5% — below random (33.3%); DeepSeek 39.0–45.6% - LLaVA and Qwen collapse to predicting "close" (Qwen ≥93% of predictions), indicating superficial alignment rather than spatial reasoning

LoRA fine-tuned models: - Best angle accuracy: LLaVA 74.35% (InterHand2.6M), MAE 0.263 - Best distance accuracy: LLaVA 90.79% (InterHand2.6M), MAE 0.094 - DeepSeek fine-tuned: angle up to 68.00%, distance up to 88.02% - Significant persistent gap on angle vs. distance (angle remains harder post-tuning)

Zero-shot transfer (Spatial-Aware VLM vs. Base VLM): - Gesture recognition: 82.89% vs. 71.86% (+10.33% absolute) - Hand-object interaction recognition: +2.63% absolute

Limitations

  • Only 7B models evaluated due to GPU constraints; larger models may behave differently.
  • Fine-tuned models retain systematic errors on joint-angle prediction; performance gains on one spatial subtask do not reliably generalize across subtasks.
  • Benchmark uses controlled MCQ format with fixed thresholds—real-world ambiguity (e.g., near-boundary poses) is largely excluded by discarding "aligned" instances.
  • Evaluation restricted to held-out splits of the same three source datasets; out-of-distribution generalization to unconstrained images is not directly measured.
  • Transfer gains on hand-object interaction (+2.63%) are modest, suggesting video-based spatial reasoning requires additional inductive biases beyond static joint geometry.

Relevance to Vision-Language Models

HandVQA provides a rigorous, controlled diagnostic for a well-known VLM weakness—fine-grained spatial grounding—at a scale and specificity not covered by prior benchmarks. The finding that base VLMs fall below random chance on distance reasoning (a 3-class problem) quantifies the depth of the failure rather than merely documenting it, directly informing what kinds of geometric supervision are missing from current pretraining. The zero-shot transfer result is significant: it establishes that injecting structured 3D geometry into VQA fine-tuning produces generalizable spatial representations, offering a concrete recipe for improving spatial reasoning in VLMs without task-specific annotation. This connects to a broader line of work (SpatialVLM, SPHERE, What'sUp) showing that general VQA accuracy masks deep geometric blindspots, and suggests part-level intra-object reasoning as an underexplored training axis for next-generation VLMs.