Adaptive Preference Arithmetic: A Personalized Agent with Adaptive Preference Arithmetic for Dynamic Preference Modeling

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AdaPA-Agent is a personalized LLM-agent framework that models dynamic preference strengths by arithmetically combining per-preference next-token distributions at inference time, weighted by alignment-estimated strength scores derived from user-agent interaction history. It avoids explicit user feedback collection and achieves consistent gains over ReAct on conversational recommendation (+18.9%) and personalized web interaction (+14.2%).

Problem

Current LLM personalization methods treat user preferences as static or require explicit user feedback to reweight them. Two concrete gaps: (i) sparse feedback makes accurate preference-strength estimation unreliable, and (ii) inserting numeric weights as text prompts suffers from natural-language ambiguity, causing misaligned generation.

Method

AdaPA-Agent has two components:

Alignment-Based Strength Estimation. For each preference \(p_i^u\), a three-stage preference chain \(C_i = (p_i^{\text{raw}}, p_i^{\text{ref}}, p_i^{\text{exp}})\) is constructed via CoT. The current interaction \(D_t^u\) is paraphrased into \(K\) augmented variants. An LLM-based scorer rates alignment between each \((C_i, D)\) pair on a 1–10 scale; raw scores are averaged over the augmented set and normalized to produce weights \(\omega_i = s_i / \sum_j s_j\).

Controllable Personalized Generation. At each autoregressive step, \(M\) independent forward passes compute per-preference next-token distributions \(P_{p_i^u}(\cdot)\). The combined distribution is: $\(P_{\text{opt}}(w_k \mid O_{t,<k}^u, D_t^u, I_t^u) = \sum_{i=1}^{M} \omega_i \cdot P_{p_i^u}(w_k \mid O_{t,<k}^u, D_t^u, C_i)\)$ This is the closed-form KL-optimal solution \(P^\star(w_k) = \text{softmax}\!\left(\sum_i \omega_i \log Q_i(w_k)\right)\), so stronger preferences dominate while weaker ones still contribute.

Key Contributions

• Reformulates preference-strength estimation as an LLM alignment-scoring problem, eliminating the need for explicit feedback.
• Dual-side augmentation (preference chains + interaction paraphrases) reduces scoring variance from lexical diversity.
• Preference arithmetic: distributional combination at token level rather than prompt-level weight injection, eliminating natural-language ambiguity.
• Validation on two distinct personalized-agent benchmarks (Reddit-Movie conversational rec, PWABench web interaction).

Results

• Conversational recommendation (Table 1, max steps=7, all tasks): AdaPA-Agent RSR 75.41% vs. ReAct 63.40%; best baseline (InteRec) 75.00%.
• Conversational recommendation (max steps=3, overall RSR): AdaPA 41.45% vs. best baseline RecMind 36.40% and ReAct 30.20%.
• PWA single-turn overall F.Acc: AdaPA 0.851 vs. ReAct 0.815; R.Acc 0.367 vs. 0.350.
• PWA multi-turn overall F.Acc: AdaPA 0.917 vs. ReAct 0.718; R.Acc 0.386 vs. 0.369.
• Abstract headline: >18.9% and >14.2% improvements over ReAct on the two respective tasks.
• AdaPA achieves best or near-best RSR across all maximum-step budgets while maintaining competitive AIR.

Limitations

• \(M\) parallel forward passes per token multiply inference cost by \(M\) (set to 2 in experiments); scaling to more preferences is expensive.
• \(K\) interaction paraphrases and LLM alignment scoring add latency before each generation turn.
• Fixed top-\(M=2\) preference selection may discard relevant lower-ranked preferences.
• Evaluation uses GPT-4o as user simulator while the backbone is Llama-3.1-8B-Instruct — results may not transfer uniformly across model families or to real users.
• Only two task domains tested; generalization to open-domain assistants is unverified.
• No analysis of failure modes when the alignment scorer misranks preference strengths.

Relevance to Harnesses / Meta-Harnesses

AdaPA-Agent is a textbook example of an inference-time orchestration harness: it wraps a base LLM and fans out \(M\) parallel decoding paths (one per preference), then merges their token-level outputs via weighted distribution arithmetic rather than string-level aggregation. This is structurally identical to the logit-combination pattern used in speculative decoding and steering-vector harnesses, making it directly relevant to anyone designing multi-call LLM pipelines where sub-agent outputs must be merged in a principled, mathematically grounded way. The dual-side augmentation module is itself a mini fan-out harness (generate \(K\) paraphrases → LLM-score each → aggregate), illustrating how nested harness patterns can be composed. The paper also highlights a key harness design tension: prompt-based control (cheap, flexible) versus distribution-level control (precise, costly), providing empirical evidence that the latter wins when preferences are fine-grained and dynamic.