Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Personalized safety in LLMs remains critically underexplored: the same response can be safe for one user and harmful for another depending on their psychological and situational context. This paper introduces PENGUIN, a 14,000-scenario benchmark for evaluating context-sensitive LLM safety, and RAISE, a training-free two-stage agent framework that uses offline MCTS planning to selectively acquire the most safety-critical user attributes before generating a response. RAISE achieves a 31.6% safety improvement over vanilla LLMs at an average cost of 2.7 user queries.

Problem¶

Existing LLM safety benchmarks evaluate context-independent metrics (factuality, toxicity, bias), assuming a universal harm standard. In practice, identical responses carry divergent risks depending on user vulnerability (e.g., an empathetic reply that relieves a stressed user can precipitate self-harm in a user with suicidal ideation). No benchmark or method systematically addresses this user-specific safety gap, and naively providing full user context is impractical due to privacy constraints and interaction cost.

Method¶

PENGUIN Benchmark: 14,000 scenarios spanning seven high-stakes domains (Life, Education, Relationship, Health, Social, Financial, Career), each paired with a structured 10-attribute user profile (age, gender, marital status, profession, economic status, education, health, mental state, self-harm history, emotion). Each scenario has a context-free and context-rich variant. Responses are scored on three dimensions—risk sensitivity, emotional empathy, and user-specific alignment—using a 5-point Likert scale, with GPT-4o as a validated proxy evaluator (\(\kappa = 0.688\), \(r = 0.92\)).

RAISE Agent Framework: Two-stage, training-free: - Offline phase: Attribute selection is formulated as a sequential decision problem. LLM-guided Monte Carlo Tree Search (MCTS) explores acquisition paths under a budget constraint, storing optimal query-path pairs for retrieval. - Online phase: A dual-module agent deploys (1) an acquisition module that retrieves the precomputed MCTS path and sequentially queries the user for the most informative attributes, and (2) an abstention module that decides whether the collected context is sufficient for safe response generation or whether to withhold/redirect.

No gradient updates are required; RAISE wraps any black-box LLM via prompting alone.

Key Contributions¶

PENGUIN: First large-scale personalized safety benchmark; 14,000 scenarios (7,000 real-world Reddit, 7,000 synthetic), 7 domains, 10-attribute structured profiles, context-rich/context-free paired variants.
Personalized safety evaluation framework: Three-dimensional metric (risk sensitivity, emotional empathy, user-specific alignment) with validated GPT-4o-as-judge reliability.
RAISE: Training-free, MCTS-guided two-stage agent that achieves 31.6% safety improvement over vanilla LLMs at ≤2.7 queries average interaction cost.
Attribute sensitivity analysis: Quantifies per-attribute contribution; psychological/situational attributes (mental state, self-harm history, emotion) contribute more than demographic ones.

Results¶

Context-free LLM safety scores: consistently 2.5–3.2/5 across all six models tested (GPT-4o, LLaMA-3.1-8B, Mistral-7B, QwQ-32B, Qwen-2.5-7B, DeepSeek-7B-chat); no model exceeds 4.0 in any domain.
Full user context (context-rich) raises average scores from 2.79 → 4.00 (+43.2%); improvement ranges from 37.5%–45.6% by domain.
RAISE vs. heuristic-based acquisition: +5.9% safety improvement.
RAISE full framework vs. vanilla LLMs: +31.6% safety improvement at 2.7 queries average interaction cost.
Psychological/situational attributes (emotion, mental state, self-harm history) are highest-impact; demographic attributes (age, economic) contribute least.

Limitations¶

GPT-4o-as-judge introduces potential evaluator bias; human annotation covers only 350/14,000 cases for validation.
MCTS offline planning assumes a fixed query budget; optimal paths may not transfer across domains without recomputation.
The 0.4% pass rate on Reddit data creates selection bias toward unusually detailed posts.
Attribute sensitivity analysis is sampled (1,000 scenarios); domain-specific optima may differ.
RAISE is evaluated on closed-domain scenarios; generalization to open-domain or multi-turn conversational contexts is not demonstrated.
Privacy implications of systematically eliciting sensitive user attributes (mental state, self-harm history) are acknowledged but not deeply analyzed.

Relevance to Harnesses / Meta-Harnesses¶

RAISE is a canonical meta-harness pattern: a training-free orchestration layer that wraps an arbitrary base LLM, using offline MCTS planning to precompute optimal multi-step information-gathering strategies and an online dual-module agent to execute them. This is structurally equivalent to a harness that intercepts user-LLM interactions, inserts a planning-driven probe sequence, evaluates sufficiency via an abstention module, and conditionally routes to response generation—all without touching model weights. The offline-plan / online-dispatch split mirrors the separation between harness configuration (strategy discovery) and runtime execution seen in meta-harness architectures. For researchers building harnesses, RAISE demonstrates how MCTS can be used to discover optimal tool/query invocation sequences under budget constraints, a technique directly applicable to any harness that must minimize costly external calls (API queries, database lookups) while maximizing downstream task quality.