Skip to content

SkillSight: Seeing Through Shared Descriptions for Accurate Skill Retrieval

🕒 Published (v1): 2026-07-21 07:05 UTC · Source: Arxiv · link

Why this paper was selected

Skill retrieval for large libraries; solves accuracy bottleneck in skill-augmented agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM agents querying large skill libraries suffer from a retrieval bias caused by shared descriptive boilerplate across skill documents, which inflates dense similarity scores for irrelevant candidates. SkillSight is a training-free calibration framework that removes this shared background in both semantic (subspace projection) and lexical (token reweighting) spaces. It improves Recall@10 by up to 20.21 percentage points over a base dense retriever while being up to 1,248× faster than neural reranking.

Problem

As agent skill libraries scale to tens of thousands of entries, retrieving the correct skill via dense retrieval degrades because skill documents share heavy structural boilerplate: capability scope, invocation interfaces, execution conditions, and usage instructions. This shared descriptive background accumulates in dense relevance scores \(s(q,d) = q^\top d\), producing an asymmetric "energy gap" — skill documents lie systematically higher in the corpus background subspace than queries (\(E_D > E_Q\)) — causing hard negatives with matching boilerplate but mismatched operations/constraints to rank above the correct skill.

Method

SkillSight decomposes retrieval bias into a semantic channel and a lexical channel, both calibrated without training:

Semantic Background Calibration (SBC). Generic tokens are identified by thresholding the IDF distribution via Otsu's method (low-IDF set \(T_g\)). For each \(t \in T_g\), the mean document embedding \(g_t = \frac{1}{|I_t|}\sum_{i \in I_t} d_i\) is computed. These token-induced directions plus the corpus mean are stacked and SVD-decomposed to obtain an orthonormal background basis \(B \in \mathbb{R}^{d \times r}\). Both query and document embeddings are projected onto the orthogonal complement: $\(q_\perp = (I - BB^\top)q, \quad d_\perp = (I - BB^\top)d\)$ giving calibrated score \(s_\perp(q,d) = q_\perp^\top d_\perp\). Rank \(r\) is selected automatically from the spectral effective rank.

Lexical Evidence Calibration (LEC). Non-generic query tokens \(T_q^{sp} = T_q \setminus T_g\) receive calibrated weights \(w_\beta(t) = \mathrm{idf}(t)\,(1 - p(t))^\beta\) where \(p(t) = \mathrm{df}(t)/N\) is document prevalence and \(\beta \geq 0\) controls suppression strength (\(\beta=1\) in practice). The lexical score is the weighted coverage of matching non-generic tokens in the candidate document.

Score fusion. Both channels are z-score normalized within a candidate pool \(C_q\) of size \(K_c = 300\), then combined as \(s_{\text{SkillSight}} = \tilde{s}_\perp + [\tilde{s}_{\text{lex}}]_+\), where the lexical term contributes only when above average, making the dense channel the primary signal.

Key Contributions

  • Systematic empirical analysis showing that shared descriptive background in skill corpora creates a pronounced query–document background energy gap absent in conventional text retrieval datasets.
  • Orthogonal decomposition \(q^\top d = (P_B q)^\top(P_B d) + (I-P_B)q^\top(I-P_B)d\) quantifying how background alignment adds spurious relevance to hard negatives.
  • SkillSight: a training-free, inference-free (beyond the base encoder) calibration framework combining SBC and LEC with automatic hyperparameter selection.
  • Demonstrated generalization across three embedding models (Qwen3, SkillRouter, SkillRet embeddings) without retraining.

Results

  • Recall@10 on SRA-Bench: 86.23% vs. 66.02% (Dense baseline), 77.13% (Dense–BM25 RRF), 77.18% (Dense + Reranker); +20.21 pp over Dense.
  • Recall@10 on SkillBench-Supp: 64.24% vs. 56.56% (Dense), 59.77% (ABTT); consistent gains across the larger 77k-skill corpus.
  • End-to-end Overall (SRA-Bench): 38.80% (Llama-3.1-8B), 69.67% (GPT-5.4-mini), 61.13% (Qwen3-4B) — best non-oracle across all three models; outperforms LLM Selection by 4.97, 1.84, and 4.28 pp respectively.
  • Latency: 1.17 ms (SRA-Bench) and 2.57 ms (SkillBench-Supp), vs. 1,460 ms (Dense + Reranker) and 167 ms (Dense–BM25 RRF); 677–1,248× speedup over reranking.
  • vs. specialized SkillRouter pipeline: +15.84 Recall@10 on SRA-Bench while being 1,397× faster; marginal −2.13 pp on SkillBench-Supp with 716× speedup.
  • Ablation: LEC alone raises Recall@10 from 66.02 to 79.76; SBC contributes complementary gains; full combination reaches 86.23.

Limitations

  • The background subspace rank \(r\) is selected via spectral effective rank, which may not be optimal under all corpus distributions; the paper notes end-to-end gains depend on dataset characteristics.
  • Recall gains on SkillBench-Supp are concentrated in top-10 coverage; MRR and Hit@5 improvements are smaller, and SkillRouter leads on Recall@10 there despite being 716× slower.
  • LEC relies on exact token overlap; skill documents in languages or domains with heavy paraphrase may not benefit from the lexical channel.
  • Evaluation uses a single embedding backbone (Qwen3-Embedding-0.6B) for the main results; behavior with very large encoders is not tested.
  • The method addresses document-side bias but not query-side ambiguity or multi-step skill chaining.

Relevance to Agentic AI / LLM Agents

Accurate skill/tool retrieval is a foundational bottleneck in any agent architecture relying on external capability libraries: a miscretrieved skill either fails silently or forces the agent to retry with inflated context. SkillSight directly improves the retrieval component of the skill-augmented agent loop and demonstrates clear end-to-end task gains across diverse benchmarks (ToolQA, MedCalc, BigCodeBench), making it practically relevant for building agents with large tool registries. The identification of "shared descriptive background" as a structural property of skill corpora — distinct from ordinary text retrieval — is a conceptual contribution that should inform future skill library design and retrieval system choices. The training-free nature and sub-millisecond latency make it drop-in compatible with existing dense-retrieval agent pipelines.