Skip to content

Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting

🕒 Published (v1): 2026-07-17 12:05 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Exo2EgoPose addresses egocentric 3D hand pose forecasting by transferring knowledge from paired exocentric (third-person) videos into an ego-view multimodal transformer, bypassing the limited field-of-view and highly dynamic motion that plague head-mounted camera streams. The framework introduces a dual-level reconstruction module that forces ego-view features to encode exo-view spatial and temporal structure during training, then injects that structure at inference via adaptive modulation. Evaluated on three benchmarks plus a robot manipulation dataset, it outperforms prior state-of-the-art and demonstrates human-to-robot transfer.

Problem

Egocentric hand pose forecasting is hampered by two intrinsic limitations of head-mounted cameras: (1) truncated field-of-view causing incomplete hand/object appearance, and (2) highly dynamic ego-motion coupling head movement with hand motion, making it difficult to disentangle hand kinematics. Prior work (OCT, USST, MADiff) relies on a single visual modality and predicts only coarse trajectories or hotspots. VLA/VRM approaches that incorporate language suffer from scarce robot data and the human–robot embodiment gap. No prior method explicitly uses complementary exocentric demonstrations to compensate for these ego-view deficiencies in a fine-grained, joint-level pose forecasting setting.

Method

The task (VL-EHPF) takes as input \(T'\) ego observation frames \(O_{t-T'+1:t} \in \mathbb{R}^{T' \times H \times W \times 3}\), a language instruction \(l\), and hand pose states \(S_{t-T'+1:t} \in \mathbb{R}^{T' \times 42 \times 3}\) (42 = 21 keypoints × 2 hands), and predicts future 3D hand keypoints \(\hat{S}_{t+1:t+\bar{T}} \in \mathbb{R}^{\bar{T} \times 42 \times 3}\).

Multimodal tokenization. Four modality-specific encoders extract features: a frozen MAE encoder (pixel-level) and DINOv2 (semantic-level) for frames, a text encoder for \(l\), and a pose encoder for \(S\). A perceiver resampler downsamples visual tokens to fixed size \(N\); all tokens are projected to hidden size \(C\) and time-embedded.

Dual-level Exocentric Reconstruction Module (DERM). A causal multimodal transformer processes ego tokens and a set of learnable queries \(\{q_p, q_v, q_f, q_e\}\) (pose, video-level Exo, frame-level Exo, ego queries). During training only, DERM uses paired exo video to define two reconstruction targets: - Video-level \(v^{exo} \in \mathbb{R}^{1 \times C_v}\): mean of per-frame MAE [CLS] tokens over the entire observation+forecast window, capturing global temporal dynamics. Loss: \(L_V = \frac{1}{C_v}\sum_d (q_v''(d) - v^{exo}(d))^2\). - Frame-level \(f^{exo} \in \mathbb{R}^{\bar{T} \times C_v}\): per-frame [CLS] tokens for the future chunk, capturing local spatial context. Loss: \(L_F = \frac{1}{\bar{T} C_v}\sum_{i,d}(q_{f,i}''(d) - f_i^{exo}(d))^2\). - An ego frame reconstruction loss \(L_E\) (MAE-style) is also included. Overall: \(L_{recon} = \lambda_V L_V + \lambda_F L_F + \lambda_E L_E\).

At inference, Exo video is absent; the DERM-decoded queries implicitly carry the learned exo structure from ego inputs.

Global-to-Local Modulation Module (GLMM). Uses the reconstructed hierarchical exo representations to refine decoded pose queries \(q_p' \in \mathbb{R}^{\bar{T} \times C}\) in two stages: - Global Modulation Module (GMM): Six modulation scalars \(\{\alpha_1, \beta_1, \gamma_1, \alpha_2, \beta_2, \gamma_2\}\) are computed from \(q_v''\) via an AMU (MLP+SiLU). Adaptive layer normalization (AdaLN) calibrates feature distribution; multi-head cross-attention between pose queries and \(q_v''\) captures long-range temporal context. - Local Modulation Module (LMM): Cross-attention between globally modulated features \(\bar{q}_m^g\) and frame-level \(q_f''\), followed by a gated residual (\(g' = \text{Sigmoid}(\text{MLP}(A_l))\)) and local AMU, performs fine-grained spatial alignment.

Final prediction: MLP over locally modulated features, supervised by smoothL1 (\(L_P\)) for pose and BCE (\(L_{va}\)) for joint validity masks.

Key Contributions

  • Formulation of the VL-EHPF task: joint prediction of future ego 3D hand poses from visual observations, language instruction, and pose state history.
  • DERM: dual-level (video + chunked frame) exocentric reconstruction as a training-time self-supervision signal that builds ego-exo cross-view correspondence without requiring Exo video at inference.
  • GLMM: hierarchical global-to-local feature modulation via AdaLN and cross-attention, injecting temporal and spatial exo knowledge progressively into ego pose features.
  • EgoMe-pose benchmark: automatic 3D hand pose labeling pipeline on top of the EgoMe dataset (82+ hours of asynchronous Ego-Exo paired videos with text annotations), providing a new evaluation setting with real-life observer/follower scenarios.
  • Demonstrated human-to-robot transfer on the CALVIN robotic manipulation benchmark.

Results

(Paper text is truncated before experiment tables; only qualitative claims are present.) - Exo2EgoPose outperforms state-of-the-art methods "by a large margin" on AssemblyHands, Ego-Exo4D, and EgoMe-pose benchmarks (no specific numerical values available in provided text). - Achieves "impressive performance improvements" on the CALVIN robot manipulation dataset, demonstrating effective human-to-robot transfer.

Limitations

  • Exo video required at training: the method depends on paired ego-exo video datasets; extending to ego-only training corpora is not addressed.
  • Asynchronous alignment is approximate: for EgoMe, temporal alignment between ego and exo streams uses linear interpolation, which may introduce misalignment errors.
  • EgoMe-pose annotations are automatic: the pose labels are produced by an automatic detection + labeling pipeline with a confidence threshold (\(\sigma_d \geq 0.5\)), introducing potential label noise.
  • No test-time Exo: while inference does not use Exo video, the reconstruction objective gives no explicit guarantee that the latent approximation quality degrades gracefully when exo coverage is imperfect or dataset Exo cameras vary widely.
  • Specific ablation evidence for each module's contribution is not visible in the provided text.

Relevance to Vision-Language Models

This paper is directly relevant to VLM researchers as it instantiates a vision-language-pose multimodal transformer in which language instructions condition fine-grained 3D motor prediction — an embodied counterpart to VLA models like RT-2. The dual-encoder visual backbone (MAE + DINOv2) and perceiver resampler tokenization reflect design choices common across modern VLMs, and the DERM's cross-view reconstruction objective is a novel auxiliary objective that could be adapted to other VLM grounding tasks requiring view-complementary supervision. More broadly, the work probes the boundary between VLMs and embodied action models: it uses language to specify intent and visual context to forecast manipulation primitives, positioning 3D hand pose as a human-interpretable, robot-transferable intermediate representation that sidesteps the scarce-data bottleneck of direct VLA training.