Skip to content

UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UFO unifies object detection, instance segmentation, semantic segmentation, and image-level vision-language tasks within a single MLLM using only an open-ended language interface—no task-specific decoders. The core novelty is reformulating segmentation as an embedding retrieval problem, where a learned <MASK> token embedding queries output image features via dot-product similarity. On GiT's multi-task benchmark, UFO outperforms the prior generalist SOTA by 12.3 mAP (COCO instance segmentation) and 3.3 mIoU (ADE20K semantic segmentation).

Problem

Integrating fine-grained perception (detection, segmentation) into generalist MLLMs requires either (a) task-specific decoders (SAM, RPN, mask heads) that add architectural complexity and complicate training, or (b) text-based approximations (polygon vertices, per-pixel class labels) that either lose detail on complex/amorphous shapes or produce unacceptably long sequences. No prior approach cleanly unifies these tasks within a standard open-ended language interface without sacrificing performance or generality.

Method

Bounding boxes: Represented as textual numbers enclosed by <box> tags (e.g., person,<box>465,268,589,344</box>), avoiding extra location tokens and remaining fully open-ended.

Segmentation via embedding retrieval: A special <MASK> token is added to the vocabulary. At inference, the model outputs this token; its hidden-state embedding em is extracted and dot-product-matched against the LLM's output image features hv (scaled by √d). Positions with positive similarity scores form the binary mask. The intuition is that MLLMs already encode spatial mask information in their output image features—retrieval decodes it without a separate decoder.

Multi-token upsampling: To overcome low-resolution image features (e.g., CLIP-L/14 downsamples by 14×), the model autoregressively predicts N²=16 <MASK> tokens. Each token corresponds to one cell of an N×N upsampling grid; its similarity map is concatenated and reshaped to produce an (Hp·N)×(Wp·N) mask—effectively learning image-conditioned upsampling parameters rather than using fixed bilinear interpolation.

Multi-task data template: Single-prediction tasks use <Prompt><Image><Response>; multi-prediction tasks (detection, instance seg) inject per-grid-point local image features as visual prompts and decode M=K² subtasks in parallel with masked self-attention between grid sequences, achieving blockwise parallel autoregressive generation.

Training: CrossEntropy loss for all tasks; focal + dice losses added for segmentation. Validated on UFO-ViT (SAM-pretrained ViT-B/L/H + BERT tokenizer) and UFO-InternVL2.5-8B. Fine-grained instruction tuning uses LoRA (rank 8) on 2.5M samples across 6 tasks.

Key Contributions

  • Decoder-free unification of object detection, instance/semantic segmentation, captioning, REC, and RES under a single open-ended language interface.
  • Embedding retrieval formulation for segmentation: <MASK> token embedding queries LLM output image features via dot product, no mask decoder required.
  • Multi-token upsampling scheme (16 <MASK> tokens default) that learns image-conditioned, per-image upsampling parameters instead of fixed interpolation.
  • Parallel multi-prediction decoding via local visual prompts and masked self-attention, enabling efficient instance-level output without sequence ordering ambiguity.
  • Demonstrated seamless integration with existing MLLMs (LLaVA-1.5, InternVL2.5) without architectural modification.

Results

  • COCO instance segmentation (GiT benchmark): UFO-ViT-H 48.1 AP vs. GiT-H 35.8 AP (+12.3 mAP); multi-task UFO-ViT-H 48.1 mAP
  • ADE20K semantic segmentation: UFO-ViT-H 55.7 mIoU vs. GiT-H 52.4 mIoU (+3.3 mIoU)
  • COCO object detection: UFO-ViT-B single-task 47.8 mAP vs. Deformable-DETR 45.4 mAP
  • Multi-task synergy: +0.9 mAP instance seg, +0.7 mIoU semantic seg, +3.1 CIDEr captioning from single→multi-task (GiT improved only +0.1 mIoU)
  • ReasonSeg (gIoU, finetuned): UFO-InternVL2.5-8B 67.0 vs. HiMTok-8B 60.8 (+6.2); vs. LISA-7B* 47.3
  • REC (RefCOCO avg): UFO-InternVL2.5-8B* 89.6 vs. VisionLLM v2 85.0; matches Ferret-v2-7B 89.3
  • RES (RefCOCO avg, cIoU): UFO-InternVL2.5-8B* 78.3 vs. SAM4MLLM-8B 76.4; matches HiMTok-8B 77.7
  • VQA retention: GQA 60.8 (vs. 60.6 base), MMBench 84.2 (vs. 84.6 base)—general ability largely preserved

Limitations

  • Beam search must be applied specifically to detection, instance segmentation, and captioning to counter class-imbalance artifacts from the large open vocabulary; this is a workaround that complicates the "fully open-ended" claim.
  • Mask resolution still depends on MLLM image feature resolution; the N=4 multi-token upsampling is a learned approximation, not a principled high-resolution decoder.
  • Parallel multi-prediction decoding requires K² grid-point forward passes, which may be slower than specialized detectors at high object densities despite parallelism.
  • ReasonSeg gains require additional reasoning segmentation data in fine-tuning; zero-shot performance (UFO-InternVL2.5-8B at 60.0 gIoU) still lags HiMTok finetuned (60.8).
  • No evaluation on video, 3D, or open-vocabulary settings beyond COCO/ADE20K categories.

Relevance to Vision-Language Models

UFO directly attacks the longstanding gap between MLLM image-level understanding and pixel-level perception by showing that output image features from a standard transformer already encode sufficient spatial information for segmentation—making task decoders architecturally unnecessary. For VLM researchers, the embedding retrieval formulation is a conceptually clean bridge: it treats the <MASK> token as a learned query over the model's own visual representation, unifying segmentation with language generation in parameter space. This enables deeper task synergy (e.g., reasoning segmentation outperforming CoT-based decoder methods) and provides a straightforward path to extending any existing MLLM with fine-grained perception via LoRA without modifying the architecture. The work establishes a new SOTA on the GiT generalist benchmark and ReasonSeg, making it a key reference for efforts to build truly general-purpose VLMs.