Skip to content

Vocabulary-Guided Gait Recognition

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

This paper proposes Gait-World, a paradigm that uses VLM text encoders to inject human-readable gait-cycle vocabulary (the eight clinical gait phases) as semantic priors into gait recognition networks. The first instantiation, α-Gait, bridges the vocabulary-to-gait modality gap with a Vocabulary Relation Mapper (VRM) and Gait Fine-grained Detector (GFD), achieving state-of-the-art results across five benchmarks without adding inference overhead.

Problem

Existing gait recognition networks (appearance-based and model-based) learn exclusively from limited gait databases, producing features that are not universal across clothing, viewpoint, and occlusion covariates. The semantic concepts researchers use to reason about gait (e.g., "loading response," "pre-swing") are disconnected from what networks actually learn, making the networks opaque and brittle. VLMs, despite their broad knowledge, cannot reliably parse fine-grained silhouette details (e.g., distinguishing individual legs), so naive VLM integration fails.

Method

Gait-World paradigm: A Vocabulary Base of eight clinically defined gait-cycle phase labels (initial contact → terminal swing) is encoded by a frozen text encoder (CLIP, LLaMA, or DeepSeek-R1-Distill). VLMs are sensitive to these global-structure phase cues even in silhouettes, so they yield meaningful vocabulary features despite being insensitive to local limb details.

α-Gait architecture: 1. A standard Gait Encoder + Horizontal Partitioning extracts per-part temporal features O ∈ ℝ^{Cg×S×P}. 2. VRM (Vocabulary Relation Mapper): Projects vocabulary features Vf via a linear + ReLU + LayerNorm transition, then applies a self-attention-style relational query (QR over aligned vocabulary keys/values) to establish inter-phase associations in the gait embedding space—intentionally omitting the linear projections in the attention to preserve the original vocabulary distribution. 3. GFD (Gait Fine-grained Detector): Treats aligned vocabulary tokens QV as DETR-style object queries and gait part features O as regions of interest; cross-attention retrieves phase-specific temporal slices from the gait sequence. 4. The resulting feature F is passed to a standard Gait Head (BNNeck + Separate FCs) trained with triplet + cross-entropy loss. At inference, the text encoder is frozen and embeddings are cached offline, adding negligible compute.

Key Contributions

  • Gait-World paradigm: first framework to bring human-defined vocabulary and VLMs into gait recognition as interpretable semantic priors.
  • α-Gait model: VRM + GFD architecture that resolves modality heterogeneity between text and silhouette embeddings without end-to-end VLM fine-tuning.
  • Vocabulary Base selection principle: empirical observation that VLMs are accurate on global gait-cycle phase labels but not local limb localisation, motivating the eight-phase clinical vocabulary as the right alignment anchor.
  • Demonstrated vocabulary generality across appearance-based, model-based, and multimodal backbone gait networks (consistent +2–2.6 pp Rank-1 gains in ablations).

Results

  • CASIA-B (α-Gait-T): 98.9% NM / 96.8% BG / 88.6% CL → 94.8% mean Rank-1; competitive with or exceeding prior SOTA (CLTD 98.6/96.4/89.3/94.8, VPNet 98.3/96.3/90.0/94.9).
  • CCPG clothing-change (α-Gait-S): 87.3% mean (Gait Eval); exceeds DeepGaitV2 (83.7%) by ~4 pp mean.
  • SUSTech1K in-the-wild (α-Gait-T): 86.3% Rank-1; exceeds DeepGaitV2 (82.3%) by ~4 pp.
  • Gait3D (α-Gait-M): 76.3% Rank-1 / 67.8% mAP; surpasses GaitMoE (73.7%/66.2%) by +2.6/+1.6 pp.
  • GREW (α-Gait-L): 81.2% Rank-1; surpasses VPNet (80.0%) by +1.2 pp.
  • Efficiency (Gait3D): 59.1 M params / 85.6 G FLOPs vs. DeepGaitV2 (25.5 M / 85.3 G / 74.4%): same compute, +1.9 pp accuracy; vs. DyGait (133.1 M / 239.0 G / 66.3%): 74 M fewer params, 153 G fewer FLOPs, +10.0 pp accuracy.
  • Ablation — Text Encoder (Gait3D Rank-1): random init 71.8% → learnable query 74.1% → CLIP 75.2% → LLaMA 74.8% → DeepSeek-R1-Distill 76.2%, showing vocabulary quality matters.

Limitations

  • Only eight vocabulary embeddings are used; the paper acknowledges richer vocabulary exploitation (e.g., attribute labels per subject) remains unexplored.
  • Does not yet constitute a true multimodal paradigm—vocabulary is a shared prior, not per-sample natural-language description.
  • Parameter count (59.1 M) is higher than lightweight baselines like GaitBase (19.3 M), though FLOPs are comparable to top performers.
  • Performance on Gait3D and GREW is constrained by upstream silhouette extraction quality, a limitation shared with all appearance-based methods but not resolved here.
  • VLMs still fail at fine-grained silhouette analysis (e.g., right-leg localisation), so only coarse phase-level vocabulary is exploitable under the current design.

Relevance to Vision-Language Models

This work is directly relevant to VLM researchers as a case study in domain-specific vocabulary grounding: it exposes that off-the-shelf VLMs (GPT-4o, CLIP) have near-zero sensitivity to fine-grained gait silhouette details while retaining useful global semantic structure, motivating selective vocabulary prompt design rather than end-to-end VLM fine-tuning. The VRM+GFD cross-modal alignment strategy—mapping frozen text encoder outputs into a specialist vision encoder's feature space via lightweight attention—is a transferable recipe for other domains where VLMs are knowledge-rich but modality-blind. It also demonstrates that VLM knowledge can serve as a universal regularizer across multiple vision backbone families (CNN-based, skeleton-based, multimodal) without architectural redesign, broadening the scope of VLM-guided representation learning beyond typical image-text tasks.