Skip to content

Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation

🕒 Published (v1): 2026-07-14 18:15 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Text2Sign is a text-conditioned video diffusion model for American Sign Language generation designed to run on a single NVIDIA L4 GPU. It combines a frozen CLIP text encoder with a 3D UNet backbone augmented by DiT-style blocks using factorized spatial-temporal attention. The system is presented explicitly as a research baseline, not a production system, and achieves only weak prompt-specific semantic control despite lower validation loss with frozen conditioning.

Problem

Training and evaluating video diffusion models for sign language production requires multi-node infrastructure, making research inaccessible to single-GPU practitioners. Additionally, naïve full spatio-temporal attention over \(T \times H \times W\) tokens scales as \(\mathcal{O}((THW)^2)\), which is prohibitive for video diffusion at practical resolutions. Prior GAN-based sign generation systems lack open code, shared benchmarks, and broader evaluation suites.

Method

The model follows DDPM with a cosine noise schedule. The backbone is a 3D UNet (base channels \(c_0 = 96\), channel multipliers \((1, 2, 4)\)) with DiT-style transformer blocks inserted at spatial resolutions \(16 \times 16\) and \(8 \times 8\). Text conditioning uses a frozen CLIP ViT-B/32 encoder producing \(z_y \in \mathbb{R}^{L \times 512}\), injected via cross-attention within each transformer block. Timestep information is injected via Adaptive Layer Normalization (AdaLN). The core efficiency contribution is factorized spatio-temporal attention: spatial attention operates on \((H, W)\) tokens per frame independently (\(\mathcal{O}(T \cdot (HW)^2)\)) followed by temporal attention along \(T\) at each spatial location (\(\mathcal{O}(HW \cdot T^2)\)), reducing total complexity from \(\mathcal{O}((THW)^2)\) to \(\mathcal{O}(T(HW)^2 + HW \cdot T^2)\). Inference uses 8-step DDIM with classifier-free guidance (\(w = 5.0\)). Training uses AdamW (\(\text{lr} = 5 \times 10^{-5}\)), EMA (\(\tau = 0.9999\)), gradient checkpointing, and AMP. The dataset is a signer-disjoint 90/10 split of How2Sign clips resized to \(64 \times 64\), \(T = 32\) frames (3,683 train / 399 val clips).

Key Contributions

  • Factorized spatial-then-temporal attention within DiT blocks, reducing attention complexity from cubic-in-volume to quadratic-per-dimension.
  • Demonstration that a frozen CLIP encoder achieves lower short-budget validation loss (0.0648) than a jointly trained 6-layer custom transformer encoder (0.0728), an 11.0% relative improvement, while reducing trainable parameters by 7.7%.
  • Systematic three-axis ablation (DiT block presence, text encoder strategy, attention factorization) on a signer-disjoint How2Sign partition.
  • A fully reproducible single-GPU baseline (NVIDIA L4, 24 GB) with public code and checkpoints.

Results

  • Best short-ablation validation loss: 0.0648 (full model, 3 epochs); longer-run checkpoint: 0.00999.
  • SSIM: \(0.2403 \pm 0.0238\); PSNR: \(15.11 \pm 0.42\) dB; Temporal consistency: \(1.0000 \pm 0.0000\) on compact evaluation slice.
  • Inference: 32-frame \(64 \times 64\) clip in 12.60 s (2.54 fps), 3.12 GB peak memory, single NVIDIA L4, 8-step DDIM.
  • Ablation (validation loss at epoch 3):
  • Full model (factorized DiT + frozen CLIP): 0.0648
  • No DiT (conv-only): 0.0805 (+19.5% vs. full)
  • Custom text encoder (jointly trained): 0.0728 (+11.0% vs. full)
  • Full 3D attention: 0.0664 (+2.5% vs. full)
  • Conditioning audit: removing text raises late-timestep denoising loss from 0.9875 to 0.9891; shuffled prompts remain nearly indistinguishable from correct prompts — indicating only weak semantic control.
  • No direct numerical comparison to prior work on a shared benchmark is reported; GAN baselines (Sreemathy et al., Kumar et al.) use different datasets and protocols.

Limitations

  • Output resolution capped at \(64 \times 64\) with 32-frame clips; not suitable for intelligible, full-resolution signing.
  • No expert linguistic evaluation or native signer preference study; SSIM/PSNR do not measure sign intelligibility.
  • Held-out conditioning audit shows only marginal separation between correct and shuffled text prompts, indicating the model has not learned robust semantic prompt control.
  • Ablations run for only 3 epochs × 50 steps; conclusions about architectural ordering may not generalize to larger compute budgets or resolutions.
  • Dataset limited to ~4K short ASL clips from How2Sign; identity-disjoint split is imperfect due to uneven identity representation.
  • No comparison on a shared benchmark against SignGen or other diffusion-based sign systems.

Relevance to Vision-Language Models

Text2Sign directly employs a VLM component — frozen CLIP (ViT-B/32) — as the semantic bridge between natural language prompts and video generation, making it a concrete case study in how pretrained VLM representations transfer to a low-data, constrained-compute generative domain. The result that frozen CLIP conditioning outperforms a jointly trained text encoder under budget constraints reinforces a recurring finding in the VLM literature: large-scale vision-language pretraining provides priors that small-data fine-tuning cannot replicate. However, the weak prompt-specific separation in the conditioning audit highlights a known challenge for VLM-conditioned generation: lower loss does not imply semantically faithful control, especially when the generative backbone lacks the capacity to exploit the full richness of the conditioning signal. This work is relevant to researchers studying VLM-conditioned video synthesis, accessibility applications of VLMs, and the efficiency-fidelity trade-offs when adapting VLM encoders to specialized low-resource domains.