Skip to content

InstanceControl: Controllable Complex Image Generation without Instance Labeling

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InstanceControl is a two-stage framework for controllable multi-instance image generation that eliminates manual instance labeling at inference time. A fine-tuned VLM automatically parses instance descriptions from text and predicts per-instance masks in the visual condition (canny/depth/HED), then a mask refinement module injects these correspondences into a FLUX diffusion transformer via a log-bias attention mask. It outperforms both labeling-free and labeling-dependent baselines on MIG-Eval and COCO-POS benchmarks.

Problem

ControlNet-style methods fail in complex multi-instance scenes because they cannot associate each textual instance description with its spatial region in the visual condition, producing attribute confusion (e.g., wrong colors, identities mixed across instances). Recent fixes require users to supply segmentation masks or bounding boxes per instance at inference time, which is labor-intensive and impractical. No prior method achieves fine-grained multi-instance control without such labeling.

Method

Stage 1 — Instance-level Text-Visual Condition Association. The system adopts Sa2VA (SAM + VLM backbone) fine-tuned on a custom 50K-image dataset. Given a text prompt \(\mathbf{p}\) and a visual condition \(\mathbf{c}\) (edge/depth/HED map), the VLM produces a structured response embedding each instance phrase between <p>…</p> tags followed by a <SEG><id> token. The Shared SEG Token (SST) strategy aggregates hidden embeddings from all <SEG> tokens sharing an instance ID: $\(\mathbf{h}_i = \text{Concat}(\mathbf{h}_i^1, \mathbf{h}_i^2, \ldots, \mathbf{h}_i^M)\)$ This unified representation is passed through SAM's decoder together with visual features \(\mathbf{f} = \mathcal{F}_{enc}(\mathbf{c})\) to yield a predicted mask \(\mathbf{m}_i^{pred}\) and confidence score \(s_i^{pred}\). Training uses cross-entropy on text plus BCE + DICE on masks.

Stage 2 — Instance-aware Controllable Generation. A lightweight U-Net mask refinement module \(f\) fuses \(\mathbf{m}_i^{pred}\), an attention-based mask \(\mathbf{m}_i^{att}\) (averaged cross-attention maps from the diffusion model), \(s_i^{pred}\), and image latent features \(\mathbf{H}_{img}\) at each denoising timestep \(t\): $\(\mathbf{m}_i^{rfn} = f(\mathbf{m}_i^{pred}, \mathbf{m}_i^{att}, s_i^{pred}, \mathbf{H}_{img})\)$ When \(s_i^{pred}\) is high, \(\mathbf{m}_i^{rfn} \approx \mathbf{m}_i^{pred}\); when low, it falls back toward \(\mathbf{m}_i^{att}\). These refined masks form a correspondence mask \(\mathbf{M} \in \mathbb{R}^{L_{img} \times L_{txt}}\) that routes image tokens to their instance text tokens: $\(\mathbf{M}[q,k] = \begin{cases} \mathbf{m}_i^{rfn} & q \in \mathcal{I}_i,\; k \in \mathcal{T}_i \\ 1 & q \in \mathcal{I}_{bg} \\ 0 & \text{otherwise} \end{cases}\)$ This is applied as a \(\log(\mathbf{M})\) additive bias in softmax attention (Eq. 8), effectively zeroing out cross-instance attention. The generation backbone is FLUX with LoRA; training uses rectified flow-matching loss plus mask BCE + DICE.

Dataset: 50K images from SAM/COCO/UniWorld-V1 with canny/depth/HED conditions; Gemini 2.5 Pro generates dense captions (avg 183.15 tokens) and instance-level noun correspondence; 1,000+ analogy-based samples (via Nano Banana image editing) for cross-domain grounding robustness.

Key Contributions

  • Labeling-free multi-instance control: eliminates per-inference instance mask/box annotation while maintaining fine-grained attribute fidelity.
  • VLM-driven instance-level grounding: adapts Sa2VA to non-RGB visual conditions (edges, depth) via a custom fine-tuning dataset with analogy-based augmentation.
  • Shared SEG Token (SST): handles instances described across multiple disjoint phrases by aggregating <SEG> hidden states under a shared instance ID.
  • Adaptive mask refinement module: dynamically blends predicted masks with diffusion attention maps conditioned on confidence scores, correcting localization noise without discarding structure.
  • Correspondence mask attention injection: encodes instance-text bindings as a log-bias mask in the FLUX Transformer MM-DiT attention, suppressing inter-instance interference.

Results

  • Evaluated on MIG-Eval (5,400 images, disjoint from training data) and COCO-POS benchmarks.
  • Metrics: MIoU (spatial alignment), Local CLIP and VQA-based Accuracy (region-wise instance fidelity via Qwen2-VL-72B), FID and ImageReward (global quality).
  • Baselines: FLUX ControlNet (no labeling), DreamRenderer, EliGen (box-text), CreatiLayout (box-text), Seg2Any (seg-text).
  • The paper states the method "outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control" across all visual conditions; specific table values are not reproduced in the provided text excerpt.
  • Qualitative figures (Fig. 1, Fig. 3) show InstanceControl correctly assigns colors/identities to numbered instances in scenes where FLUX ControlNet and labeling-based methods produce red-boxed errors.

Limitations

  • Stage 1 VLM can produce noisy or incomplete masks (acknowledged; motivates the refinement module but not fully eliminated).
  • The approach requires fine-tuning on a specially constructed 50K dataset with expensive Gemini 2.5 Pro annotations; generalization to out-of-distribution visual conditions or languages is untested.
  • The SST strategy and VLM grounding can fail when text is semantically inconsistent with the visual condition modality (e.g., cartoon characters described over human edge maps), partially addressed by analogy augmentation but acknowledged as a challenge.
  • Mask refinement U-Net adds per-timestep overhead during the denoising loop; computational cost not quantified in the provided text.
  • Evaluation omits CLIP Score and PickScore due to the 77-token prompt limit, so standard cross-paper comparisons on those metrics are unavailable.

Relevance to Vision-Language Models

InstanceControl exemplifies using VLMs not as generators but as structured grounding modules: it fine-tunes Sa2VA (SAM+VLM) to parse free-form text into instance descriptions and segment them within non-RGB condition maps, a challenging cross-modal reasoning task where existing RGB-trained models fail. The work directly extends VLM grounding research (LISA, Sa2VA, GLaMM) to a new application domain — visual condition modalities — by constructing aligned training data where text and image inputs are intentionally semantically mismatched. The confidence-gated fusion between VLM-predicted masks and diffusion attention maps is a practical recipe for combining VLM spatial reasoning with generative model internal signals, a paradigm increasingly relevant as VLMs are integrated into diffusion pipelines.