Language-Assisted Super-Resolution from Real-World Low-Resolution Patches¶

🕒 Published (v1): 2026-06-30 08:57 UTC · Source: Arxiv · link

Why this paper was selected

Kyoung Mu Lee; language-guided real-world SR; creative cross-modal approach

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LA-SR proposes an unpaired super-resolution framework that exploits camera-subject depth variation within a single image to obtain real LR-HR patches, then uses CLIP-based linguistic-content and linguistic-quality losses to bridge the domain gap without paired supervision. The method outperforms both unsupervised and supervised SR baselines on perceptual metrics across real-world and synthetic benchmarks.

Problem¶

Synthetic degradations (bicubic downsampling, ISP simulation) fail to capture the full complexity of real-world image degradation, causing SR models to generalize poorly in practice. Existing unpaired/self-supervised methods either constrain LR-HR patches to share near-identical textures (DGDML-SR) or require specialized camera hardware, neither of which is realistic or scalable.

Method¶

Depth-guided patch extraction: Uses a pre-trained depth estimator to select distant (low-resolution) regions as \(I_{LR}\) and close (high-resolution) regions as \(I_{HR}\) from the same high-quality image, with no requirement that they share similar content.
CLIP-based encoders: Pre-trained CLIP image encoder \(E_I\) (with positional embeddings removed for variable-size input) and text encoder \(E_T\) project patches and text into a shared space.
Linguistic-content loss (\(\mathcal{L}_{LC}\)): Contrastive learning aligns each patch with its corresponding content text (generated by a pre-trained image-to-text model). \(I_{SR}\) is trained to correlate with \(I_{LR}\)'s content text, preserving semantics.
Linguistic-quality loss (\(\mathcal{L}_{LQ}\)): Contrastive learning separates \(I_{LR}\) from \(I_{HR}\) in quality space using binary quality texts (e.g., "good" vs "bad"), then trains \(I_{SR}\) to align with the high-quality text distribution.
Modified VGG loss: Applied between \(I_{SR}\) and \(I_{LR}\) with additional pooling to handle resolution mismatch, preserving low-frequency structure/color.
Training: RRDB backbone (ESRGAN) initialized from PSNR-oriented pre-trained weights, then fine-tuned with \(\mathcal{L}_{tot}=0.5\mathcal{L}_{LC}+0.5\mathcal{L}_{LQ}+0.1\mathcal{L}_{VGG}\) for 300k iterations on 4×Quadro RTX 8000 (50 hours).

Key Contributions¶

First unpaired SR framework that trains on real (not synthetic) LR patches extracted from high-quality images via depth cues
Reformulates unpaired SR as a language-space alignment problem, using two novel CLIP-based losses to separately control content preservation and perceptual quality
Demonstrates superior perceptual quality across 7+ benchmarks against both unsupervised and supervised SR methods, including generalization to AI-generated images

Results¶

DRealSR (real-world LR): LA-SR achieves BRISQUE 30.07, CLIPIQA 0.46, TOPIQ 0.40, MUSIQ 40.70 — surpassing DASR, PDD, and SRTTA (Table 1).
CameraFusion: BRISQUE 12.47, CLIPIQA 0.69, TOPIQ 0.58, MUSIQ 58.97 — beats MASA-SR and DCSR (Table 1).
CUFED5: BRISQUE 5.70, CLIPIQA 0.66, TOPIQ 0.61, MUSIQ 67.26 (Table 1).
Supervised comparison (Set5/Set14/BSD100/Urban100/DIV2K): LA-SR consistently beats ESRGAN (bicubic-trained), ESRGAN† (camera-LR-trained), and RealESRGAN (synthetic-degradation-trained) on all four perceptual metrics (Table 2).
AI-generated images: On Dataset Diffusion and LDM samples, LA-SR maintains superior perceptual quality over all supervised baselines (Table 2, bottom).
Depth-based patch selection: Outperforms overlap/inclusion/random patch selection (Table 3); BRISQUE 10.90 vs 13.72/11.76/8.45 on DIV2K.
Training uses \(56\times56\) \(I_{LR}\) and \(224\times224\) \(I_{HR}\) with scaling factor \(s=4\).

Limitations¶

No explicit LR-HR pairs used for content supervision — text descriptions may misrepresent fine visual details when the captioning model is imprecise.
Quality loss relies on binary quality text (good/bad) — a continuous quality representation might better capture fine-grained realism.
50-hour training on 4 GPUs is computationally heavy; inference cost inherits CLIP encoder overhead.
Main evaluation centers on no-reference perceptual metrics; distortion metrics (PSNR/SSIM) are relegated to the appendix and likely weaker.
Depth estimation errors (especially on complex scenes with occlusions or textureless regions) can produce poorly matched LR-HR patches.

Relevance to Vision-Language Models¶

LA-SR demonstrates a novel application of CLIP beyond classification/retrieval: using the joint embedding space to provide domain-alignment supervision for a low-level vision task. It shows that VLMs can separate what an image contains (content) from how detailed it is (quality) in the same latent space, enabling language-guided loss functions that replace paired training data. This is part of a growing trend where VLMs serve as perceptual priors for image restoration and generation, rather than just as understanding-only backbones.