Skip to content

MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MEgoHand is a multimodal framework for egocentric hand-object interaction (HOI) motion generation that conditions on RGB images, text instructions, and an initial MANO hand pose. It pairs a VLM-based "cerebrum" for semantic and spatial reasoning with a DiT flow-matching "cerebellum" for trajectory generation, achieving 86.9% wrist translation error reduction and 34.1% joint rotation error reduction over the strongest baseline.

Problem

Existing egocentric HOI generation methods rely on predefined 3D object priors (geometry, mass), restricting generalization to novel objects. Multimodal alternatives suffer from ambiguous text-only guidance, complex contact-map pipelines, and open-loop accumulation of viewpoint/occlusion errors. Dataset heterogeneity (inconsistent language annotations, pose representations, missing depth) further obstructs unified training.

Method

MEgoHand uses a bi-level architecture:

High-level (Cerebrum): Eagle-2 VLM (SmolLM2 language backbone + SigLIP-2 vision encoder) processes text instructions and RGB frames. A UniDepthV2 monocular metric depth estimator produces a dense depth map encoded by a ResNet-50 depth encoder; additive fusion combines RGB and depth features before cross-modal attention with text embeddings, yielding a joint representation z^{TDI} that captures semantics, spatial geometry, and task intent — without any object-specific priors.

Low-level (Cerebellum): A Diffusion Transformer (DiT) conditioned on z^{TDI} and the initial MANO parameters h_k generates future MANO parameter sequences via conditional flow matching. Overlapping motion chunks are produced and decoded with Temporal Orthogonal Filtering (TOF): rotations from overlapping predictions are temporally averaged then projected onto the SO(3) manifold via SVD to suppress jitter without training overhead.

Dataset curation: An Inverse MANO Retargeting Network recovers MANO parameters from raw joint coordinates using a two-stage gated training (shape first, then wrist pose). A Virtual RGB-D Renderer synthesizes aligned depth maps for RGB-only datasets. Eight datasets (TACO, HOT3D, HOI4D, H2O, FPHA, OakInk2, ARCTIC, HOLO) are unified into 3.35M RGB-D frames, 24K trajectories, 1.2K objects.

Key Contributions

  • First framework to use a VLM for motion prior inference in egocentric HOI, replacing object-specific models with visual-textual reasoning.
  • Monocular metric depth integration enabling object-agnostic 3D spatial understanding.
  • Temporal Orthogonal Filtering (TOF): a training-free SO(3)-respecting decoding strategy for temporal coherence.
  • Inverse MANO Retargeting Network with a two-stage shape-then-pose training strategy for cross-dataset pose normalization.
  • Virtual RGB-D Renderer for synthesizing paired depth maps from 3D meshes, enabling depth-supervised training on RGB-only datasets.
  • Unified 3.35M-frame multimodal HOI dataset spanning 1.2K objects and 24K interaction trajectories.

Results

  • In-domain (5 datasets — TACO, HOI4D, H2O, HOT3D, OakInk2):
  • MWTE: 4.756 cm vs. LatentAct 7.221 cm → 34.1% reduction; vs. LatentAct 7.221 → 86.9% MRE reduction (0.123 rad vs. 0.937 rad)
  • MPJPE-PA: 0.425 cm vs. 1.478 cm (LatentAct) → 71.2% improvement
  • MPVE-PA: 0.409 cm vs. 1.453 cm → 71.9% improvement
  • Cross-domain (ARCTIC): MPJPE 7.358 cm vs. LatentAct-Diff 10.98 cm → 33.9% improvement
  • Cross-domain (HOLO): MPJPE 5.775 cm vs. LatentAct-Diff 8.235 cm → 29.8% improvement
  • Ablations confirm depth supervision is necessary (removal raises MPJPE by ~5.5%), and metric depth outperforms relative depth in-domain (though cross-domain sensitivity increases).
  • MEgoHand-ID (no text) already achieves 50% lower MPJPE than LatentAct, while removing contact maps from LatentAct degrades it by 10.3%.

Limitations

  • Only right-hand motion is addressed in this submission; bimanual scenarios are excluded.
  • Training data consists primarily of rigid objects; ARCTIC (articulated objects, e.g., scissors) reveals a performance gap under complex dynamic coupling.
  • The inverse MANO retargeting network and virtual renderer require 3D mesh annotations — datasets with only 2D or sensor-based annotations still need preprocessing.
  • Metric depth input is sensitive to cross-domain camera parameter variation, partially reducing its cross-domain advantage over relative depth.
  • Scale of annotatable in-the-wild data remains limited; broader coverage via modern detectors is listed as future work but not yet demonstrated.

Relevance to Vision-Language Models

MEgoHand is a concrete instance of using a frozen/fine-tuned VLM backbone (Eagle-2) as a motion-prior extractor rather than a text-output model, demonstrating that VLMs can ground abstract instructions into geometric action priors for continuous 3D trajectory generation. It highlights a systematic challenge for VLMs in embodied settings: pre-trained visual encoders excel at 2D semantics but underperform on 3D spatial reasoning, necessitating auxiliary depth pathways. The paper is directly relevant to VLM-for-robotics and embodied-AI threads, showing how multimodal fusion (vision + language + depth) enables generalization to novel objects without task-specific 3D priors — a paradigm applicable beyond hand-object interaction to broader manipulation and AR/VR grounding tasks.