HMVLM:Human Motion-Vision-Language Model via MoE LoRA

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HMVLM is a unified multimodal framework that integrates 3D human motion, vision, and language using a Mixture-of-Experts LoRA (MoE LoRA) fine-tuning strategy built on top of a pretrained LLM. A novel "zero expert" — a non-trainable, zero-initialized LoRA pair — is introduced to preserve the foundation model's world knowledge during instruction tuning across heterogeneous motion tasks. Body-part-specific tokenization via spatial Transformers and per-part VQ-VAE codebooks replaces temporal-only tokenization, improving spatial granularity for both static poses and motion sequences.

Problem

Two unresolved issues in integrating 3D human motion into foundation language models: (1) catastrophic forgetting — instruction tuning on motion tokens causes severe degradation of pre-trained linguistic capabilities (e.g., MotionAgent drops 87.16% on MT-Bench); (2) coarse pose representation — standard temporally-compressed VQ-VAE tokenization lacks spatial expressiveness, making single-frame pose tasks (e.g., pose estimation) heavily constrained by codebook size.

Method

• MoE LoRA: The pretrained weight update is W' = W + Σ αᵢ AᵢBᵢ over n experts. A CLIP text encoder processes task instructions and prompts; a two-layer MLP gating network outputs per-expert mixture weights α. During inference, LoRA experts are dynamically blended — no pre-merging is possible.
• Zero expert: Expert 0 has frozen zero-initialized matrices A₀, B₀. A gating loss L_gat = -E[η · log p(α₀|I,P)] (η=1 for non-motion tasks) pushes α₀ → 1 for general dialogue, effectively bypassing LoRA and preserving pretrained weights.
• Body-part tokenization: A spatial Transformer encodes each joint with learnable body-part parameters and part-constrained attention masks, pooling outputs per body part into N part embeddings. Each part has an independent codebook; for motion sequences, a temporal convolution module further compresses across frames. Reconstruction loss combines L_rec + L_emb + λ_com · L_com.
• Tasks covered: text-to-motion (T2M), monocular 3D pose estimation (image input), motion video understanding (8-frame uniform sampling via CLIP ViT-L/14 + LLaVA projection). Base model: Vicuna-7b-v1.5 (primary), also tested on Gemma-2-2b-it.

Key Contributions

• MoE LoRA framework with a zero expert for multitask multimodal fine-tuning that decouples motion tasks from general language tasks.
• L_gat gating loss that explicitly trains the router to select the zero expert on non-motion prompts, enabling knowledge-preserving instruction tuning.
• Body-part-specific VQ-VAE tokenizer using a spatial Transformer with per-part independent codebooks, extending standard temporal-only tokenization to the spatial domain.
• Unified model supporting T2M generation, 3D pose estimation, and motion video understanding within a single LLM backbone.

Results

• Knowledge preservation (MT-Bench, 80 questions, GPT-4 judge):
• MotionGPT (Llama2): 22.71% drop (2.73 → 2.11)
• MotionAgent (Gemma2): 87.16% drop (7.79 → 1.00)
• Ours (Gemma2): 3.34% drop (7.79 → 7.53)
• Ours (Vicuna): 6.10% drop (5.90 → 5.54)
• Ours w/o L_gat (Vicuna): collapses to 1.00 across all categories
• T2M on HumanML3D (single-task setting): Top-1/2/3 R-precision 0.502/0.692/0.785, FID 0.123, MM-D 3.039 — competitive with or better than multimodal baselines (MotionGPT, MotionAgent, MotionGPT-2); below task-specific SOTA MoMask (FID 0.045).
• Pose estimation (H3.6M): MPJPE 92.8 / PA-MPJPE 55.3 (single-task) vs. ChatPose 126.0 / 82.4.
• Body-part tokenization ablation: Top-3 R-precision 0.785 vs. 0.741 (whole-body K=512) and 0.758 (whole-body K=512×5); reconstruction MSE 0.966 vs. 1.377 and 1.34.
• Gating weights: Zero expert weight ≈ 0.999 for general dialogue; drops to 0.694 (T2M), 0.454 (pose), 0.167 (video understanding).

Limitations

• Modality connections are learned in an independent pairwise manner; no holistic cross-modal integration.
• Domain discrepancies across datasets (HumanML3D, KIT-ML, Human3.6M, 3DPW, MoVid) hinder any-to-any generation.
• MoE LoRA prevents pre-merging expert weights at inference, causing moderate latency increase as expert count scales.
• Multi-task T2M performance degrades compared to single-task (Top-1 R-precision: 0.463 vs. 0.502), due to shared parameter budget across tasks.
• Pose estimation still lags behind specialized regression-based methods (SPIN: MPJPE 61.9, HMR 2.0: 50.0).

Relevance to Vision-Language Models

HMVLM is directly relevant to the growing body of work on extending VLMs to non-visual modalities, specifically 3D human motion, while preserving the core language capabilities of the foundation model. The zero-expert MoE LoRA strategy offers a principled solution to catastrophic forgetting — a pervasive concern whenever new modality tokens are injected into a pretrained LLM's vocabulary — with quantitative evidence rarely shown in prior motion-language work. The body-part tokenization approach parallels patch-based spatial encoding in vision (ViT), suggesting a broadly applicable principle for structured-domain tokenization (skeleton, graph, mesh) within autoregressive LLM frameworks. For VLM researchers, the paper demonstrates that task-routing via gating over LoRA experts is effective at preventing task interference in unified multimodal models without architectural surgery.