Miss-ReID: Delivering Robust Multi-Modality Object Re-Identification Despite Missing Modalities¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Miss-ReID is a multi-modality object Re-ID framework that handles missing visual modalities (RGB/NIR/TIR) during both training and inference by using CLIP's vision-text alignment to generate compensatory textual embeddings for absent modalities. It is the first work to jointly address modality-missing conditions at both training and inference time. On RGBNT201, it achieves 54.6% mean mAP across six missing-modality scenarios, outperforming prior SOTA by 3.9% mAP.

Problem¶

Existing multi-modality ReID models assume modality completeness at both training and inference. When any modality is absent at inference (due to sensor failure, privacy constraints, etc.), performance degrades sharply—e.g., TOP-ReID drops 63.8% mAP when RGB+TIR are missing. Prior reconstruction-based approaches (DENet, TOP-ReID) require complete training data and cannot handle missingness during training.

Method¶

Miss-ReID uses CLIP's frozen visual and text encoders and three collaborative modules:

M-bHIPR (Memory-based Heterogeneous Identity Prototype Representation): Extracts per-modality CLS features via a CLIP image encoder. Maintains three separate modality-specific memory banks, each storing per-identity prototype vectors initialized as feature centroids and updated via EMA (α=0.2). Optimized with ClusterNCE contrastive loss against stored prototypes.
M-iOSM (Modality-invariant Object Structure Modeling): Uses Nq=16 learnable "structural probe" query vectors in a cross-attention mechanism over local patch embeddings from available modalities. Queries attend to patches via Multi-Head Attention, then averages across queries and averages across available modalities to produce a single modality-invariant structural feature vector.
L-dMMC (Language-driven Missing Modality Completion): Maps the structural feature to a pseudo-word token via an MLP inversion network. Concatenates this pseudo-word with learnable modality-specific prompts (Np=4 per modality) and inserts into a text template ("An image of a [pseudo-word] person, who shows the [modality m] attributes.") fed into the frozen CLIP text encoder. The resulting textual embedding substitutes the missing visual modality's feature. A memory-based text-to-vision contrastive loss (LT2V) aligns textual embeddings to visual prototypes in the memory bank.

Modality missingness is simulated during training by randomly zeroing modality inputs with probability ηm. Total loss combines triplet + label-smoothing CE + ClusterNCE + LT2V across visual, structural, and textual features.

Key Contributions¶

First framework supporting modality-missing conditions at both training and inference for multi-modality ReID.
M-bHIPR: modality-specific memory banks with EMA prototype updates, preventing cross-modality feature contamination.
M-iOSM: learnable structural probe queries extract modality-invariant spatial structure from available patches.
L-dMMC: textual inversion of structural features into CLIP-compatible pseudo-word tokens with modality-specific prompts, enabling VLM-driven compensation for missing visual modalities.
Memory-based text-vision contrastive alignment (LT2V) to close the vision-text semantic gap at the prototype level.

Results¶

RGBNT201, modality-complete training + missing inference: Miss-ReID achieves 54.6% mean mAP / 55.7% mean R-1 across six missing scenarios vs. DeMo (50.7%/50.8%), IDEA (50.5%/–), TOP-ReID (44.4%/45.4%).
Lowest performance drop: In the RGB-only-missing (R̄NT) scenario, Miss-ReID drops only 13.4% mAP vs. TOP-ReID's 24.8%; in the hardest two-modality-missing case (R̄N̄T), achieves 34.5% mAP vs. TOP-ReID's 26.2%.
Ablation (RGBNT201, η=0.1): Full model (F) gains +5.7% mean mAP / +5.3% mean R-1 over the baseline. Each module contributes positively; L-dMMC alone yields +1.0% mAP; combining all three gives the largest gain.
Robustness to missing rate: Mean mAP degrades from 54.6% at η=(0,0,0) to 47.3% at η=(0.5,0.5,0.5), remaining above PCB (19.7%) and TOP-ReID (44.4%) trained on complete data.
Parameter overhead: 89.6M params / 43.6G FLOPs vs. baseline 86.4M / 34.3G (M-bHIPR and M-iOSM add no inference-time cost).

Limitations¶

Performance still degrades substantially under extreme cases (e.g., two missing modalities: 34.5% vs. 76.9% mAP in fully available case).
Does not address complete modality collapse (all modalities missing simultaneously).
Evaluated only on RGB/NIR/TIR; no extension to other modality types (event cameras, LiDAR, audio, sketch).
Text encoder is frozen; the inversion network must bridge the visual-to-text gap without fine-tuning the underlying language model.

Relevance to Vision-Language Models¶

Miss-ReID demonstrates a practical use of CLIP's vision-text alignment beyond classification and retrieval—specifically using it as a semantic proxy for missing visual modalities in a structured retrieval task. The textual inversion approach (converting visual structural features into pseudo-word tokens) is a concrete method for injecting task-specific visual semantics into frozen VLM text encoders, a technique of broad interest for VLM adaptation. The paper also highlights that VLMs' cross-modal alignment is strong enough to substitute missing modalities in a fine-grained identity-matching task, extending VLM applicability to sensor-incomplete real-world pipelines. For researchers tracking VLMs, this work sits at the intersection of prompt tuning, textual inversion, and multi-modal learning under distribution shift.