APeB: Benchmarking Personalization Ability of Large Language Model Agents¶

🕒 Published (v1): 2026-07-03 10:03 UTC · Source: Arxiv · link

Why this paper was selected

Benchmark for LLM agent personalization; evaluation frameworks drive field standards

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

APeB (Agent Personalized Benchmark) is a benchmark for evaluating LLM agent personalization under realistic conditions: underspecified user queries, heterogeneous behavioral histories, and closely competing product candidates. Testing on 5,648 cases shows that multi-step agent workflows (ReAct, Deerflow) improve semantic retrieval on refined queries but fail to help—and sometimes hurt—when users issue vague, early-stage intent queries. A lightweight history-grounded pipeline, VQRA, partially closes the gap by making history use explicit.

Problem¶

Existing personalization benchmarks each address at most one or two axes of the challenge in isolation—either they use sanitized histories, user-refined queries, or randomly sampled candidates—and none jointly require an agent to infer latent user intent from noisy heterogeneous histories while selecting among closely competing alternatives. This makes it impossible to evaluate whether agentic pipelines (e.g., ReAct) genuinely contribute to preference-aligned decisions under real-world ambiguity.

Method¶

APeB is constructed from action logs of a content-shopping platform. Sessions are extracted as 30-minute search-to-order windows satisfying three criteria: iterative query refinement (\(|Q^\text{search}_u| \geq 2\)), extensive browsing (\(|P^{\text{view}}_u| + |V^{\text{view}}_u| \geq \tau_v\)), and semantic intent–order alignment (verified by GPT-4.1 with human checks). Each case contains: - Heterogeneous history \(H_u\): product and media (video/livestream) interactions prior to the session, with per-record metadata and dwell time. - Two query types: intent (earliest aligned query, vague/early-stage) and refined (last pre-purchase query, explicit). - Hard candidate set \(C^{\text{hard}}_u\): browsed-but-unbought items plus the ground-truth purchase; candidates are close semantic competitors.

Evaluation uses a structured output protocol (Algorithm 1): models produce user-target inference, history summarization ranked by relevance, and ranked recommendations. Outcome metric is Hit@1/5. Rubric-based LLM-as-a-Judge scores intermediate steps across Query Inference (QI), Preference Quality (PQ), Recommendation Quality (RQ), and Agent Reasoning (AR) dimensions. Three agent workflows are tested: Single-Prompt, ReAct, Deerflow, and VQRA (a history-grounded vague query rewriter that preprocesses the intent query before ranking).

Key Contributions¶

First benchmark combining heterogeneous noisy histories, underspecified intent queries, and hard-negative candidate sets in a single task.
Non-trivial session filtering pipeline grounded in real behavioral logs (5,648 cases, 10+ categories, avg. 187 interactions/user).
Empirical evidence that ReAct workflows improve refined-query performance but are neutral or harmful on intent queries.
Rubric decomposition identifying intent inference failure and ineffective history attribution as the primary performance bottlenecks.
VQRA, a minimal history-aware query rewriting wrapper that consistently raises intent Hit@1 across all tested LLMs.

Results¶

Intent queries (hard candidates): Single-prompt LLMs Hit@1 = 22.1–25.6%; supervised UniSAR = 25.9% — LLMs match but do not exceed a trained retrieval baseline.
Refined queries (hard candidates): LLMs Hit@1 = 33.9–43.3%; UniSAR = 29.2% — LLMs substantially outperform; ReAct further boosts top models (e.g., GPT-5.2Think 37.4% → 43.3%).
Random candidates: LLMs achieve 84.8–95.9% Hit@1 vs UniSAR 81.3%, showing strong semantic matching when personalization is not required.
History ablation: Removing history drops intent Hit@1 by up to 9.0% (R1: 24.2→22.2); UniSAR gains +84% from 60 category-prior records vs. no-history on intent queries; LLMs show far smaller relative gains.
VQRA: Raises intent Hit@1 by 0.3–2.7 points over single-prompt across all models tested.
Deerflow: Yields no consistent benefit; matches or underperforms single-prompt on both query types.

Limitations¶

Dataset sourced from a single content-shopping platform (ByteDance); generalizability to other domains is unverified.
History limited to product and media interactions; other modalities (search history, social signals) are excluded.
LLM-as-a-Judge rubric evaluation introduces potential circularity when the same model family (GPT-4.1, GPT-5.2) is both evaluated and used as judge.
VQRA uses a shared GPT-4.1 backbone, making it an unfair comparison for models without access to that backbone.
Text is truncated; full analysis of AR rubric results and the Deerflow failure mode is not available in the provided excerpt.
APeB is planned for controlled release pending compliance review; not yet publicly accessible.

Relevance to Harnesses / Meta-Harnesses¶

APeB functions as a benchmark harness for evaluating multi-step agentic pipelines: it wraps LLMs inside structured workflows (ReAct, Deerflow, VQRA) and measures end-to-end task completion alongside rubric-graded intermediate reasoning—essentially a meta-evaluation harness that instruments the interior of agent pipelines, not just final outputs. The VQRA component is itself a lightweight meta-harness: a preprocessing pipeline that wraps any base LLM to improve history utilization before the downstream agent runs. The finding that off-the-shelf agentic frameworks (ReAct, Deerflow) fail to help on intent queries unless history is explicitly managed is directly actionable for harness designers: harness architecture—not just the underlying model—determines whether retrieved context is actually used. This connects to the broader line of work showing that evaluation harnesses must expose intermediate reasoning states (rubric scores over QI, PQ, RQ) to diagnose where pipeline-level failures occur.