Skip to content

Talking Points: Describing and Localizing Pixels

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Talking Points: pixel-precise keypoint comprehension via natural language

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Talking Points introduces a bidirectional pixel-level grounding framework consisting of a Point Descriptor (image+keypoint → free-form natural language) and a Point Localizer (description → pixel coordinates), pushing VLM grounding from object/region granularity down to individual pixels. They synthesize a 20K+ dataset (LlamaPointInPart) of image–keypoint–description triplets and fine-tune the descriptor via GRPO with the localizer as a reward model for cross-category generalization. The method doubles the OMG-LLaVA baseline and outperforms ChatGPT-5 and human annotators on the new benchmark.

Problem

Existing VLMs and grounding models (SAM, GroundingDINO, OMG-LLaVA) operate at object or segment scale; pixel-precise keypoint grounding via free-form language is unsupported. Prior keypoint-language models (KptLLM, LocLLM) rely on rigid, template-based descriptions tied to predefined anatomy (e.g., "left shoulder of human"), preventing generalizable pixel-level language grounding.

Method

Architecture. The Point Descriptor adapts OMG-LLaVA by replacing learned segmentation masks with fixed Gaussian attention masks \(M\) centered at the input keypoint \((x, y)\). Within OMG-Seg's 9 transformer decoder layers, cross-attention is restricted to image features inside the Gaussian region, yielding keypoint-specific "keypoint features" and "region features." The full image is still provided to the LLM, preserving scene context. Descriptions follow a hierarchical coarse-to-fine structure: (1) object location in image, (2) part location within object, (3) keypoint position within part, (4) local visual cues.

The Point Localizer inverts this: given image \(I\) and description \(D\), a single forward pass through OMG-LLaVA extracts the hidden state \(h \in \mathbb{R}^d\) of a special <SEG> token, which is passed through a text-to-vision projection and MLP to regress normalized coordinates \((\hat{x}, \hat{y}) \in [0,1]^2\). Training minimizes \(\mathcal{L}_{loc} = \text{MSE}(\hat{p}, p_{gt})\).

Dataset (LlamaPointInPart). 20K+ triplets constructed from PascalPart116, ADE20KPart234, and PartImageNet. SIFT selects the highest-response keypoint within annotated semantic parts. Two VLMs provide multi-scale context: LLaVA processes a Gaussian-masked crop for fine-grained local features; OMG-LLaVA processes the full image for object-centric context. LLaMA3.3 (quantized) synthesizes both into coherent hierarchical descriptions. Split: 17K train / 4K test; 64 object and 297 part categories.

GRPO RL for cross-category generalization. On AP-10K (which lacks descriptions), the frozen Point Localizer serves as the reward model. \(G\) descriptions \(\{o_i\}\) are sampled from the descriptor policy \(\pi_\theta\); reward is \(r_i = -\text{MSE}(\hat{p}_i, p)\). Normalized group-relative advantages \(\hat{A}_i = (r_i - \text{mean}(r))/\text{std}(r)\) drive a length-normalized policy gradient: $\(\mathcal{L}_{\text{policy}} = -\frac{1}{G}\sum_{i=1}^{G}\hat{A}_i \cdot \frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\log\pi_\theta(o_{i,t}|o_{i,<t}, I, p)\)$ KL regularization against a reference policy uses the unbiased estimator \([\exp(r_t) - r_t - 1]\) with log-ratio clamped to \([-5, 5]\). Only LoRA adapters in the final two transformer blocks are updated.

Key Contributions

  • LlamaPointInPart: 20K+ image–keypoint–description triplets with coarse-to-fine hierarchical descriptions, spanning 64 object and 297 part categories.
  • Point Descriptor: Free-form, pixel-level keypoint description via Gaussian masked attention in OMG-LLaVA's decoder, eliminating template constraints.
  • Point Localizer: Coordinate regression from natural language descriptions via <SEG> hidden-state MLP.
  • GRPO-based closed-loop RL: Uses the frozen localizer as a verifiable reward signal to adapt the descriptor to novel categories without ground-truth descriptions.
  • Evaluation protocol: Descriptor quality measured through localization accuracy (mPCK = average of PCK@0.1 and PCK@0.2), bypassing the need for reference text comparison.

Results

  • vs. OMG-LLaVA baseline: Talking Points (TP) achieves approximately 2× the mPCK of OMG-LLaVA on LlamaPointInPart.
  • vs. ChatGPT-5: TP outperforms ChatGPT-5 on pixel-level localization from descriptions.
  • vs. Human annotations: Human-generated descriptions perform worse than ChatGPT-5, underscoring the inherent difficulty of pixel-level language grounding.
  • Gaussian mask ablation: Removing Gaussian masks causes mPCK to collapse from 78.13 to 23.63, demonstrating that pixel-specific masked attention is essential.
  • AP-10K cross-category: Bidirectional evaluation on Bovidae↔Canidae transfer; RL fine-tuning via GRPO with localizer-as-reward enables generalization to unseen taxonomic groups.

Limitations

  • Dataset is synthetically constructed using VLMs; 91% of test samples were verified manually (9% may contain noisy descriptions).
  • SIFT keypoint selection within parts may still bias toward textured regions despite the within-part constraint.
  • Cross-category RL generalization is evaluated only on two AP-10K super-categories (Bovidae, Canidae); breadth of transfer is unverified.
  • The evaluation metric (descriptor quality via localization accuracy) creates a circular dependency: quality is measured only through the specific trained localizer, not through independent human judgment.
  • No ablation comparing GRPO to supervised fine-tuning on AP-10K when synthetic descriptions are unavailable.

Relevance to Vision-Language Models

This work directly advances the frontier of VLM spatial grounding by demonstrating that pixel-precise, free-form language grounding is achievable—something no prior VLM achieved beyond object/region granularity. The Gaussian masked-attention modification to OMG-LLaVA is a lightweight architectural change with large empirical impact, suggesting a practical recipe for adapting existing region-grounding VLMs to finer spatial scales. The closed-loop GRPO framework, using a trained model as a verifiable reward signal rather than human preferences, is a compelling template for fine-grained spatial RL applicable to other VLM tasks (GUI interaction, medical imaging, robotics). The finding that human annotators underperform ChatGPT-5 at pixel-level description also establishes a useful calibration point for future VLM grounding benchmarks.