Skip to content

Let RGB Be the Language of Vision

🕒 Published (v1): 2026-07-14 07:25 UTC · Source: Arxiv · link

Why this paper was selected

Unified RGB formulation for all visual modalities (masks, depth, etc.); architectural simplification with broad VLM implications

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RINO (RGB In and RGB Out) proposes treating all visual information — masks, depth maps, pose skeletons, surface normals — as RGB images, unifying diverse vision tasks under a single pretrained image editing backbone without any task-specific modules. The paradigm mirrors how LLMs use text as a universal interface: any understanding task becomes image→RGB-structured-output, and any conditioned generation task becomes RGB-structured-input→image. Applied zero-shot to 20+ benchmarks, RINO achieves competitive performance against in-domain specialist models.

Problem

Vision lacks a universal representation interface analogous to text in LLMs. Each task (segmentation, depth, pose, etc.) requires its own representation format, dedicated encoder/decoder, and adapter components, preventing a single model from freely transferring across tasks. This forces most vision foundation models to remain task-specific experts rather than general-purpose visual learners.

Method

RINO formalizes all vision tasks as a RGB/text-to-RGB/text editing problem using a frozen pretrained multimodal image editing model \(F\):

\[y_{\text{img}}, y_{\text{text}} = F(x_{\text{img}}, x_{\text{text}})\]

where \(F\) is a large instruction-following image editor (Qwen-Image-Edit at 20B parameters, LongCat-Image-Edit at 6B, or FireRed-Image-Edit). No task-specific parameters, heads, or adapters are added or fine-tuned.

  • Understanding: natural image in → RGB-encoded prediction out (e.g., grayscale depth visualization, colored normal map, color-coded segmentation mask). A lightweight parameter-free converter post-hoc decodes the RGB output into task-specific format (color-to-class lookup, luminance-to-depth decoding).
  • Generation: RGB-formatted condition in (pose skeleton, depth map, edge map, segmentation mask) → natural image out, treating generation as the dual of understanding.
  • Segmentation specifics: hierarchical two-stage strategy for large label spaces (10 superclasses then fine-grained); local majority filtering and small-component removal for artifact mitigation.
  • Pose specifics: color-agnostic decoder using foreground distance-transform peaks matched to a canonical skeleton template, robust to palette variation.

Key Contributions

  • Formulation of all vision understanding and generation tasks as a unified RGB/text-to-RGB/text paradigm (RINO), directly analogous to text universality in LLMs.
  • Demonstration that a single frozen image editor (no task-specific fine-tuning) achieves competitive zero-shot performance across 20+ tasks including depth, normals, semantic/instance/panoptic segmentation, object detection, human pose estimation, and conditioned generation.
  • New zero-shot state of the art on conditioned image generation (pose-to-image, depth-to-image, etc.) under existing evaluation protocols.
  • Parameter-free pre/post-inference converters that bridge RGB and task-specific formats without touching model weights.
  • Human preference studies confirming visual quality of RGB-unified outputs.

Results

  • Depth estimation (disparity-space affine-invariant): Qwen-Image-Edit achieves \(\delta_1 = 0.938\) on DIODE-indoor vs. Depth Anything V2 at \(0.952\); NYUv2 \(\delta_1 = 0.927\) vs. specialist \(0.981\).
  • Surface normals (NYUv2): FireRed-Image-Edit reaches mean angular error \(17.73°\), approaching DSINE specialist at \(16.4°\); median \(10.99°\) vs. DSINE \(8.4°\).
  • Semantic segmentation (Pascal VOC, zero-shot): Qwen-Image-Edit mAcc \(83.53\), aAcc \(85.17\), surpassing zero-shot specialists (FC-CLIP mAcc \(65.42\), aAcc \(78.46\)).
  • Semantic segmentation (ADE20K 10-class coarse): Qwen-Image-Edit mIoU \(40.37\) vs. FC-CLIP \(42.17\) and MaskCLIP \(39.26\).
  • Object detection / instance segmentation (COCO val2017, oracle-class/count): Qwen-Image-Edit box AP50 \(16.3\%\), mask AP50 \(13.9\%\) (vs. supervised Mask R-CNN \(63.5\%\) / \(60.5\%\)).
  • Panoptic segmentation: Qwen-Image-Edit SQ \(64.8\)–\(74.3\) across Cityscapes/ADE20K/COCO (strong spatial quality), but PQ \(6.9\)–\(13.6\) due to low RQ (recognition/instance-matching failure).
  • 2D pose estimation (COCO val2017, PA PCK@0.2): FireRed \(0.795\), Qwen \(0.774\), LongCat \(0.767\) vs. ViTPose \(0.984\); raw PCK@0.05 remains substantially below specialists (\(0.047\)–\(0.233\) vs. \(0.730\)–\(0.868\)).

Limitations

  • Performance gap versus in-domain specialist models persists on most tasks, especially at fine-grained levels (ADE20K 150-class, instance/panoptic recognition, precise keypoint localization).
  • Zero-shot transfer relies on web-scale pretraining data of the backbone editors, which likely contains diverse visual signals (masks, depth maps) but at very low frequency — this incidental coverage is hard to control or audit.
  • Instance-level tasks (detection, panoptic things) suffer from over-detection, over-segmentation, and inter-instance confusion fundamental to the per-class silhouette decoding strategy.
  • Panoptic recognition quality (RQ) is severely limited: Cityscapes things PQ = \(1.5\) for Qwen, indicating near-complete failure on instance association.
  • Affine-invariant depth evaluation hides metric scale errors; linear depth-space evaluation reveals larger performance drops on wide-range scenes (KITTI, iBims-1).
  • RINO-Zero has no explicit training signal for non-natural visual data; the full RINO with instruction tuning is left to future work.

Relevance to Vision-Language Models

RINO directly operationalizes the analogy between text-as-universal-interface in LLMs and RGB-as-universal-interface in vision, positioning large multimodal image editors (themselves VLMs coupling a vision-language encoder with a diffusion decoder) as the backbone for general-purpose visual understanding. The work demonstrates that VLM-based image editors implicitly encode rich structured visual knowledge — depth, normals, segmentation semantics — in their latent space, recoverable purely through natural-language prompting and RGB decoding. This has implications for unified vision-language systems: it suggests that scaling instruction-tuned image editing models may subsume many specialist vision heads, reducing the gap between language-model-style task transfer and dense visual prediction. The paper provides a concrete bridge between the generative VLM paradigm (InstructPix2Pix, Qwen-Image-Edit) and classical vision benchmarks.