Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models¶

🕒 Published (v1): 2026-07-01 02:56 UTC · Source: Arxiv · link

Why this paper was selected

In-context inference for personalized object localization — few-shot VLM grounding without finetuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

IPLoc-ID extends the prior IPLoc VLM-based personalized object localization method by adding an instance-level identification component that can reject query images not containing the reference object. The key mechanism is a self-posed query inserted between a candidate bounding-box generation step and a binary Yes/No identification answer, all generated autoregressively in a single sequence. This enables the model to jointly learn reference-conditioned localization and instance-level negative-query rejection via LoRA fine-tuning.

Problem¶

IPLoc, the prior state-of-the-art for personalized object localization (POL), assumes the target object is always present in the query image. Its output space is restricted to bounding boxes in $\mathcal{B}$, so it always returns a box even for negative queries—i.e., $f_{\text{IPLoc}}(x) \in \mathcal{B}$ even when $f^*(x) = \emptyset$. This produces systematic false-positive detections, making IPLoc impractical for image retrieval, video grounding, and object re-identification, where most candidate images are negative.

Method¶

The authors define Personalized Object Identification and Localization (POIL) as learning the ideal mapping $f^* : \mathcal{X} \rightarrow \mathcal{B} \cup \{\emptyset\}$, where the model must return a bounding box only if the reference object instance is present ($\delta(x) = 1$) and reject otherwise.

IPLoc-ID solves POIL by extending IPLoc's sequence-generation paradigm with a three-part autoregressive output: $$y = \langle B \rangle \langle Q \rangle \langle A \rangle$$ where $B$ is a candidate bounding box, $Q$ is a fixed self-posed query ("Do all these boxes have the same object?"), and $A \in \{\text{Yes}, \text{No}\}$ is the identification answer. The factorized generation probability is: $$p_\theta(y | x) = p_\theta(B | x) \cdot p_\theta(A | x, B, Q)$$ since $Q$ is deterministic. The self-posed query bridges the BBOX candidate and final answer within a single token stream, requiring no extra stochastic step.

Fine-tuning uses LoRA on the frozen VLM backbone, minimizing negative log-likelihood over both positive examples (target: correct BBOX + Yes) and negative examples (target: most-plausible candidate BBOX + No). For negative examples, a plausible candidate box is still generated and used as the localization target so the model learns candidate localization even when the answer is rejection.

Datasets are constructed from four public video object tracking sources (LaSOT, PDM/Burst, GOT-10K, VastTrack) by sampling $N+1$ frames per video for reference+positive query, plus one negative query from a different sub-class or class instance.

Key Contributions¶

Formulation of the POIL task, which extends POL to include instance-level negative-query rejection.
IPLoc-ID algorithm: a single autoregressive framework that jointly produces candidate localization, a self-posed bridging query, and a binary identification answer.
POIL datasets constructed from four public video tracking benchmarks, including in-class adversarial negative examples.
Empirical demonstration that IPLoc-ID substantially reduces false-positive detections on negative queries while preserving localization accuracy comparable to IPLoc.

Results¶

IPLoc-ID substantially suppresses false-positive detections on negative query images across four evaluation datasets (LaSOT test, PDM, GOT-10K, VastTrack) compared to IPLoc, Florence-2, NT3 (No-Time-To-Train), and Qwen2-VL.
F1-score for instance-level identification improves markedly over IPLoc and conventional FSOD baselines (VFM, NT3) which lack explicit negative-rejection mechanisms.
mIoU localization accuracy on positive examples remains comparable to the IPLoc baseline, confirming the identification component does not degrade localization.
Ablation studies confirm the self-posed query is critical: removing it degrades identification performance.

(Note: the paper text provided is truncated before the quantitative results tables; specific numerical values are not available in the supplied text.)

Limitations¶

Fine-tuning is performed only on LaSOT-derived training data; generalization to very different domains relies on the VLM backbone's prior knowledge.
The self-posed query $Q$ is fixed ("Do all these boxes have the same object?"), which may not elicit optimal reasoning for all object types or reference formats.
Negative examples for GOT-10K and PDM are sampled from different classes (not same-class distractors), which is an easier regime than in-class negatives used for LaSOT evaluation.
The method still requires LoRA fine-tuning; it is not purely training-free at inference time (unlike NT3).
The paper text is truncated, so reported quantitative comparisons could not be fully verified.

Relevance to Vision-Language Models¶

This paper is directly relevant to the growing line of work using VLMs for structured visual output generation (bounding boxes, grounding) via in-context learning and next-token prediction. It extends the IPLoc paradigm—which demonstrated that VLMs can localize personalized object instances through sequence completion—by adding a natural-language identification head that reuses the same autoregressive backbone, showing how VLMs can be steered toward rejection decisions without a separate classifier. The self-posed query technique is a lightweight prompt-engineering strategy that elicits Yes/No verdicts from the generative VLM without any architectural modification, which is broadly applicable to other VLM-based detection/grounding systems. For VLM researchers, this work highlights open-set instance discrimination (as opposed to category-level detection) as a key capability gap that fine-tuning and clever prompt formulation can partially close.