Skip to content

LVLM-Driven Attribute-Aware Modeling for Visible-Infrared Person Re-Identification

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LVLM-AAM uses a large vision-language model (Qwen2.5-VL) to extract five-value attribute arrays (gender, glasses, backpack, upper/lower clothing) from unlabeled training images, then leverages these attributes to refine pseudo-labels and generate modality-bridging text features for unsupervised visible-infrared person re-identification. The method closes much of the gap between unsupervised and supervised VI-ReID without manual identity annotations, surpassing several supervised baselines on standard benchmarks.

Problem

Unsupervised VI-ReID relies on clustering-derived pseudo-labels whose quality is constrained by the pretrained image encoder and is prone to cascading inter-modality matching errors. Existing CLIP-based UVI-ReID methods (e.g., CCLNet) construct texts from the same noisy pseudo-labels—providing no fundamentally new supervision—and use text semantics only within each modality, failing to enforce modality-invariant learning across the visible/infrared gap.

Method

The pipeline has three main stages applied to an unlabeled visible+infrared training set:

  1. Attribute-Aware Reliable Labeling (ARL): Qwen2.5-VL is prompted once (offline) to extract a 5-tuple [gender, glasses, backpack, upper, lower] per image. Attribute-aware refinement (AR) removes cluster outliers whose image-level attributes differ from the cluster-level mode array by more than η=2 positions, improving intra-modality pseudo-label quality (measured by F-score). Attribute-aware matching (AM) partitions all clusters into 8 groups by the first three binary/categorical attributes and runs progressive graph matching within each group, reducing inter-modality confusion.

  2. Explicit-Implicit Attribute Fusion (EAF): Each cluster is assigned a text prompt of the form "An image of a [X1]…[XM] [male/female] [wearing/without] glasses and [carrying/without] a backpack." Learnable embeddings [Xi] (implicit) are combined with hard attribute tokens (explicit). A frozen CLIP text+image encoder pair optimizes the learnable embeddings via image-to-text contrastive loss, producing static text features.

  3. Attribute-Aware Contrastive Learning (AAC): Dynamic text features are computed as a convex combination of matched-pair static features across modalities: \(\hat{t}^v_p = (1-\alpha)t^v_p + \alpha t^r_p\) (α=0.5). The image encoder is then optimized with a text-semantic contrastive loss that pulls image features toward their dynamic (modality-mixed) text feature, promoting modality-invariant representations.

Training runs in three sequential phases: 50 epochs of intra-modality contrastive learning (ResNet-50), 50 epochs of CLIP text embedding optimization, then 50 epochs of full heterogeneous learning with Ltotal = Lintra + 0.5·Linter + 0.5·Ltsc.

Key Contributions

  • LVLM-AAM framework: first use of LVLM-extracted attribute arrays to simultaneously improve pseudo-label quality and cross-modal text semantics in UVI-ReID.
  • ARL strategy: attribute-aware refinement (outlier removal by attribute deviation) + attribute-aware matching (attribute-grouped inter-modality graph matching).
  • EAF module: combines learnable CLIP prompt embeddings with explicit hard attribute tokens for richer identity-related text representations.
  • AAC module: dynamic text features interpolated across modalities enable modality-invariant contrastive supervision without additional annotations.

Results

SYSU-MM01 All Search (vs. best UVI-ReID baseline SDCL): - Rank-1: 66.58% vs. 64.49% (+2.09%), mAP: 63.50% vs. 63.24% (+0.26%), mINP: 52.29% vs. 51.06% (+1.23%) - Outperforms supervised methods DDAG, AGW, MCLNet, FMCNet, DART on this split

RegDB Visible→Thermal (vs. best UVI-ReID baseline DLM): - Rank-1: 90.25% vs. 87.55% (+2.70%), mAP: 83.96% vs. 82.83% (+1.13%), mINP: 73.98% vs. 71.93% (+2.05%) - Rank-1 comparable to supervised SAAI and STAR-ReID

LLCM Visible→Infrared (vs. SDCL): - Rank-1: 52.2% vs. 46.9% (+5.3%), mAP: 57.3% vs. 52.4% (+4.9%) - Outperforms camera-label-aided SCA-RCP

Ablation (SYSU-MM01 All Search, Rank-1/mAP/mINP): - Baseline: 58.52/52.89/35.27 - +AR: 60.85/56.03/41.26 - +AM: 61.36/57.40/43.18 - +EAF: 62.51/59.26/46.11 - +AAC only: 64.59/61.89/50.69 - Full LVLM-AAM: 66.58/63.50/52.29

Limitations

  • Not strictly unsupervised: LVLM inference constitutes a form of external supervision (attribute labels), though no manual identity annotation is required.
  • Attribute-aware matching hard-codes 8 groups based on first three binary attributes; more flexible or learned grouping strategies are unexplored.
  • Attribute scope is narrow (5 coarse attributes: gender, glasses, backpack, upper/lower clothing); broader or finer attribute sets were not systematically studied.
  • Image encoder backbone is ResNet-50, which is weaker than Transformer-based backbones used by recent SVI-ReID state-of-the-art methods, potentially understating the gap.

Relevance to Vision-Language Models

This work demonstrates a practical paradigm for using LVLMs as offline annotation tools—replacing expensive human labeling with structured attribute extraction—and then integrating those attributes into CLIP-based prompt learning for cross-modal retrieval. It is directly relevant to VLM researchers interested in prompt engineering (explicit+implicit token fusion), zero-annotation adaptation of CLIP to specialized domains, and modality-bridging via text-feature interpolation. The approach also highlights how LVLMs can improve the training signal quality in downstream tasks beyond direct inference-time use, connecting to broader themes of LVLM-assisted pseudo-labeling and unsupervised domain adaptation with VLMs.