FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FOCUS is an end-to-end Large Vision-Language Model that unifies referential segmentation and controllable image/video editing within a single architecture, using jointly optimized segmentation masks as spatial conditioning for a diffusion decoder. Unlike prior work that chains separate segmentation and editing modules, FOCUS achieves tight cross-modal coupling through a dual-branch visual encoder, MoVQGAN-based tokenizer, and a progressive four-stage training pipeline. It achieves competitive performance on multimodal understanding, referring segmentation (RefCOCO/+/g), and image generation/editing benchmarks at the 3B scale.

Problem

Existing approaches to instruction-driven image editing either (a) chain disjoint segmentation and diffusion models with no joint optimization, (b) use LLMs as task routers over frozen expert tools without deep feature fusion, or (c) build unified perception-generation models that only support coarse text-driven control and lack object-level, mask-guided editing. No prior method jointly learns "what to edit" (referential segmentation) and "how to edit" (diffusion generation) in a single end-to-end framework with shared feature representations.

Method

FOCUS is built on Qwen2.5-3B and has four components:

1. Dual-branch visual encoder: A QwenViT processes 256×256 images for global semantics; a ConvNeXt-L processes 768×768 images for fine-grained multi-scale features. The two branches are fused via cross-attention, then injected into the LLM through a gated cross-attention adapter (3-scale: f2, f3, f4) with learnable scaling parameters initialized to zero.

2. MoVQGAN-based generative tokenizer: Two branches — a semantic branch (28× downsample, codebook 32,768) and a pixel branch (16× downsample, codebook 98,304). Crucially, continuous pre-quantization features (not discrete tokens) are fed to the LLM to reduce quantization information loss; the LLM autoregressively predicts discrete tokens in semantic-first order.

3. Segmentation decoder: Takes LLM mask token embeddings, task prompt embeddings, and multi-scale ConvNeXt features to predict binary masks, category scores, and instance embeddings (for video temporal association).

4. Diffusion decoder (SDXL-initialized): Segmentation masks are downsampled to latent resolution, linearly projected into a spatial guidance sequence, and injected via cross-attention into the UNet denoising process. Semantic and pixel tokens from the LLM condition the denoising.

Training: Four progressive stages — (1) tokenizer + diffusion decoder pretraining on 58M image pairs; (2) visual-language adapter warmup (frozen encoders); (3) joint LLM + mask decoder multimodal pretraining with segmentation-aware alignment (256→512 resolution); (4) instruction tuning for region-controlled editing (512→1024 with bucketed aspect ratio cropping).

Key Contributions

• Unified end-to-end LVLM that jointly models referential segmentation and mask-conditioned image/video editing, eliminating disjoint pipelines.
• Dual-branch visual encoder combining QwenViT (global, low-res) and ConvNeXt-L (fine-grained, high-res) fused via gated cross-attention with three feature scales.
• MoVQGAN tokenizer augmented by feeding continuous pre-quantization features to the LLM, mitigating discrete quantization information loss.
• Progressive four-stage training that escalates resolution and task complexity, preventing the hallucination degradation observed at uniformly high resolution.
• Segmentation mask injection into SDXL diffusion decoder via cross-attention for spatially precise, region-controllable editing.

Results

Multimodal understanding (Table 1, 3B model): - POPE: 88.0; MMBench: 81.5; SEED: 73.9 — competitive with Janus-Pro-7B (88.0 / 80.8 / 72.1) at 2.3× fewer parameters; outperforms ILLUME+ (3B) on all general benchmarks.

Image generation (Table 2): - MJHQ-30K FID: 6.05 (vs. Janus-Pro-7B 6.00, ILLUME+ 6.05 — 3B); GenAI-Bench Advanced: 0.72 (highest among 3B unified models); GenEval Single Obj: 1.00.

Referring segmentation (Table 3, mIoU): - RefCOCO testA/testB: 86.3 / 82.7; RefCOCO+ testA/testB: 84.1 / 74.3; RefCOCOg val/test: 79.3 / 79.8 — competitive with HyperSeg (84.8 / 85.7 / 83.4), a segmentation specialist, while also handling generation.

Image editing (Table 4, Emu Edit benchmark): - CLIP-T: 0.278 (best among all methods); CLIP-I: 0.876; DINO: 0.831 — surpasses specialized editing models (InstructPix2Pix, Emu Edit) and unified models (ILLUME+).

Ablations: - Progressive resolution (256→1024) vs. fixed 1024: FID 6.05 vs. 6.03, but POPE 88.0 vs. 77.6 — progressive training avoids hallucination degradation. - 3-scale gated cross-attention vs. single-scale: RefCOCO mIoU 81.4 vs. 79.5 / 78.2. - Continuous visual input vs. discrete only: MMBench 70.9 vs. 50.4; FID 6.05 vs. 6.85.

Limitations

• Segmentation quality on gRefCOCO (generalized referring, multi-object) lags behind top specialists (52.5 vs. not directly compared), suggesting difficulty with ambiguous multi-referent cases.
• Training is expensive: tokenizer/diffusion decoder ≈3 days each; 3B MLLM ≈13 days on a cluster — not easily reproduced.
• Video editing is frame-by-frame with no explicit temporal consistency modeling beyond instance embedding propagation, which may degrade on long clips or fast motion.
• Hallucination increases at high resolution; the progressive training mitigates but does not eliminate this, as evidenced by the fixed-1024 POPE drop.
• No ablation of the SDXL diffusion backbone size or its contribution relative to the LLM — unclear if a lighter decoder would degrade editing fidelity.

Relevance to Vision-Language Models

FOCUS directly addresses a central challenge in the VLM field: closing the loop between language-guided visual understanding and pixel-level generative control, which most LVLMs leave to external tools. The architecture demonstrates that joint training of segmentation and generation within one LLM backbone is not only feasible but mutually reinforcing — segmentation masks improve generation localization, and generation training sharpens the feature representations available to the segmenter. For researchers tracking VLMs, the key insight is the treatment of continuous pre-quantization features as LLM input (bypassing discrete bottlenecks) and the use of mask tokens as a structured intermediate representation bridging perception and synthesis — a pattern likely to influence future unified multimodal architectures. The competitive 3B results versus 7B baselines also reinforce the trend that architectural quality (multi-scale encoding, tight cross-modal fusion) can compensate for scale.