Skip to content

H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks

🕒 Published (v1): 2026-06-24 08:52 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; attention-derived masks for pose-robust image editing with VLM-adjacent mechanisms

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

H-Adapter is a diffusion-based hairstyle transfer method that trains IP-Adapter with a region-specific loss to spatially disentangle hair and non-hair objectives. This induces structured cross-attention maps from which a source-aligned hair mask is automatically derived, guiding inpainting under large source–reference head-pose discrepancies. It achieves best FID, FIDCLIP, and CLIP-I on pose-different pairs while supporting text-guided color control and identity-preserving composition.

Problem

Existing hairstyle transfer methods fail under large head-pose gaps between source and reference images due to two failure modes: (1) inaccurate hair-region localization leading to misplaced or over-extended edits, and (2) limited reproduction of fine-grained reference hair texture/structure. GAN-based methods rely on aligned face datasets; diffusion-based methods apply conditioning globally, causing non-hair content contamination and pose-inconsistent hair placement.

Method

H-Adapter fine-tunes IP-Adapter-Plus on Stable Diffusion v1.5 using a region-specific training objective on (target image, reference image, binary hair mask) triplets from FFHQ and CelebV-HQ. The objective has two terms:

\[L = L_{\mathrm{hair}} + \lambda_{\mathrm{non\text{-}hair}}\,L_{\mathrm{non\text{-}hair}}\]

\(L_{\mathrm{hair}}\) applies the standard denoising loss only inside the hair mask \(M\); \(L_{\mathrm{non\text{-}hair}}\) regularizes the adapter's prediction outside \(M\) toward the unmodified diffusion baseline \(\epsilon_{\theta_0}\) (\(\lambda=0.1\)). This yields cross-attention maps where a single token \(t_8\) (empirically identified as the "separator") attends to non-hair regions, while all other tokens attend to hair. At inference, the aggregated attention map (all tokens except \(t_8\)) is binarized into a coarse inpainting mask \(M_{\mathrm{attn}}\).

Two-stage coarse-to-fine inference: (1) A warm-up step with an all-one mask extracts \(M_{\mathrm{attn}}\); ~20 denoising steps produce an intermediate image \(y_{\mathrm{mid}}\). (2) A segmentation model extracts a pixel-level hair mask \(M_{\mathrm{seg}}\) from \(y_{\mathrm{mid}}\); full denoising from \(z_T\) with \(M_{\mathrm{seg}}\) yields the final output \(y_{\mathrm{fine}}\). Reference injection is further gated per-timestep via a spatial mask \(M_t\) in cross-attention:

\[Z = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V + \lambda\Bigl(M_t \odot \mathrm{softmax}\!\left(\frac{QK'^\top}{\sqrt{d}}\right)V'\Bigr)\]

Hair removal uses a pretrained FLUX.2 instruction editor; the segmentation model is SAM or equivalent.

Key Contributions

  • Region-specific loss (\(L_{\mathrm{hair}} + \lambda L_{\mathrm{non\text{-}hair}}\)) that spatially disentangles hair/non-hair conditioning in IP-Adapter cross-attention.
  • Attention-derived, source-aligned coarse hair mask extracted from the induced separator token \(t_8\), enabling pose-robust inpainting without explicit pose cues or latent alignment.
  • Two-stage coarse-to-fine inference pipeline progressing from attention mask → segmentation mask for pixel-level editing.
  • Per-timestep reference gating via \(M_t\) to dynamically restrict adapter influence to hair regions during denoising.
  • VLM-as-a-judge (GPT-4o) evaluation protocol scoring Hairstyle Faithfulness (HFS), Non-hair Preservation (NPS), and Artifact Quality (AQS).
  • Plug-and-play extensions: text-to-image generation, auxiliary prompt-based hair color control (via \(\lambda\) weighting), and composition with identity-preserving IP-Adapter variants.

Results

Evaluated on CelebA-HQ with 3,000 source–reference pairs; pose-different subset requires absolute yaw difference >15°.

  • FID: Ours 12.47 (best) vs. IP-Adapter(t2) 12.53, HairFastGAN 12.78, FLUX.2 12.66
  • FIDCLIP: Ours 3.98 (best) vs. FLUX.2 5.18, IP-Adapter(t2) 4.26, HairFastGAN 4.53
  • CLIP-I: Ours 0.659 (best) vs. IP-Adapter(t2) 0.651, HairFastGAN 0.649
  • SSIM: Ours 0.831 (second-best); FLUX.2 leads at 0.904
  • PSNR: Ours 23.06; FLUX.2 leads at 25.35
  • VLM-as-a-judge (GPT-4o, pose-different): Ours leads all methods on HFS (3.11), NPS (4.23), and AQS (3.73) with non-overlapping 95% CIs vs. most baselines
  • Separator token analysis: \(t_8\) achieves highest mean IoUmax over 3,000 self-pair samples across all 16 candidate tokens

Limitations

  • Coarse attention mask provides only rough localization; pixel-level accuracy depends on the intermediate segmentation step, which can fail on unusual hair textures or lighting.
  • Hair removal relies on a pretrained instruction-based editor (FLUX.2), introducing an additional dependency and potential artifacts in the base image \(\tilde{x}_s\).
  • Built on SD v1.5, limiting generation resolution and fidelity compared to modern large-scale models (e.g., SDXL, FLUX.2 natively).
  • Separator token \(t_8\) is identified empirically and may not generalize to other adapter architectures or token counts.
  • Training requires binary hair masks, adding a segmentation preprocessing dependency.
  • Evaluation is limited to frontal/near-frontal face imagery (CelebA-HQ); extreme profiles or occlusions are not systematically evaluated.

Relevance to Vision-Language Models

H-Adapter introduces a VLM-as-a-judge evaluation protocol using GPT-4o to score hairstyle faithfulness, non-hair preservation, and artifact quality — a methodological contribution directly relevant to VLM researchers studying automatic perceptual evaluation. The work also uses CLIP image embeddings (CLIP-I, FIDCLIP) as a backbone metric, connecting to broader VLM representation research. Beyond evaluation, the paper demonstrates that cross-attention structure in diffusion adapters can be leveraged for spatial grounding without explicit language prompts, which parallels attention-based localization research in vision-language grounding. The VLM-judged quality gains complement pixel-level metrics, illustrating how language-vision models are increasingly used as surrogate annotators for fine-grained visual tasks.