Skip to content

Can We Steer the Black-Box? Towards Controllability-Centric Evaluation of Recommender Systems with Collaborative Agents

🕒 Published (v1): 2026-07-15 03:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CtrlBench-Rec is a multi-agent evaluation harness that operationalizes controllability — the capacity to steer a black-box recommender system toward explicit goals — via three progressive tasks. It evolves an initial pool of \(K_{\text{init}}\) agent probes into a compact set of \(K_{\text{super}} < K_{\text{init}}\) "super probes" through BERT-based clustering and LLM-driven collaborative fusion, achieving better coverage with far fewer interactions than unguided baselines. Experiments across five recommendation architectures on MovieLens-1M and Amazon Toys & Games expose a hard ceiling: even the best method saturates at ~55% target coverage under extended budgets.

Problem

Black-box recommender systems offer no handle for users or regulators to steer outputs toward specific intents. Existing evaluation paradigms measure only predictive accuracy or apply static post-hoc analyses (interpretability, fairness); they do not quantify dynamic steerability. Single-agent RL approaches cannot accommodate diverse, immediate control goals under tight interaction budgets, and naïvely scaling agent count shifts cost without improving precision. There is no standardized, reproducible benchmark for controllability.

Method

CtrlBench-Rec is a three-stage evolutionary multi-agent framework:

  1. Initialization (Stage 1): Extract static attributes and dynamic trajectory \(H\) from a real user dataset to instantiate \(K_{\text{init}}\) agents. Each agent has a User Profile Expert composing four modules: static profile, dynamic memory, an LLM decision engine, and a tool-calling module.

  2. Interaction & alignment (Stage 2): A cold-start register→search→click pipeline synchronizes each agent's external behavioral history with the recommender's internal user index. Subsequent multi-turn cycles record actions \(a_t\) chosen from the recommendation list \(L_t\) into both the system index and agent-local memory, maintaining dual-synchronized state.

  3. Fusion & evolution (Stage 3): Each agent's profile and trajectory are encoded via BERT into \(v_i\); K-Means partitions the population into behavioral clusters. Within each cluster, agents conduct a structured multi-agent discussion; a User Profile Merge Expert synthesizes a consolidated super-profile and instantiates it as a new super probe. The process iterates until convergence, reducing \(K_{\text{init}}\) agents to \(K_{\text{super}}\) elite probes.

Three evaluation tasks probe steerability at increasing depth: - Task 1 — Target Content Discovery: maximize unique discovered items from a predefined target set \(T\) within budget \(N\). - Task 2 — Interest Profile Shaping: minimize distance between the recommended distribution and a target interest vector \(v_{\text{target}}\) (measured via cosine similarity). - Task 3 — Popularity Bias Mitigation: minimize average popularity of recommended items over the interaction horizon.

The framework is built on the Oasis simulation platform.

Key Contributions

  • First standardized multi-level controllability benchmark (target discovery → profile shaping → bias mitigation) for black-box recommender systems.
  • Evolutionary multi-agent fusion algorithm that condenses \(K_{\text{init}}=100\) random agents into \(K_{\text{super}} \approx 26\) super probes via BERT + K-Means + LLM deliberation, achieving superior coverage-per-agent.
  • Empirical identification of two systemic bottlenecks: (a) a hard representation-sparsity ceiling (~55% max coverage) and (b) a medium-popularity semantic-dispersion trap.
  • Released codebase and datasets enabling reproducible controllability auditing.

Results

  • Task 1 — Coverage at \(t=20\), BGE/Amazon: CtrlBench-Rec achieves ~22.5% vs. Base-Small (12.5%), a ~1.8× gain with equal agent count.
  • Exploration efficiency (NARM/MovieLens): CtrlBench-Rec 13.0 vs. Base-Large 25.0 (lower is better — cost per new unique item).
  • Extended budget (\(t=200\)): asymptotic coverage saturates at ~55% for CtrlBench-Rec; no method reaches full coverage.
  • Task 2 — Multi-interest shaping: final similarity \(S_{\text{final}} > 0.72\) after 20 rounds; trajectory smoothness >0.972 across all scenario levels and time windows.
  • Task 3 — Popularity bias: CtrlBench-Rec accelerates long-tail discovery at \(t < 20\) but all methods stall post-\(t=40\), confirming a systemic bottleneck.
  • Gains are consistent across five architectures (NARM, SASRec, TwHIN-BERT, BGE, Qwen3.5-4B) on both datasets.

Limitations

  • No external baselines exist; all comparisons are internal ablations, limiting cross-study validation.
  • Hard coverage ceiling (~55%) indicates ~45% of target items are inaccessible to any probe-level strategy due to representation sparsity — a property of the recommender, not the framework.
  • Long-tail items (e.g., Film-Noir: 44 items, 6.8% coverage) lie at semantic periphery and resist steering at any budget.
  • Popularity bias is a systemic failure mode that probe optimization alone cannot overcome; structural changes to the recommender are required.
  • Cold-start alignment (\(L_{\text{init}} \in [4,6]\) required) imposes an irreducible upfront interaction cost.
  • Evaluation assumes agents can inject arbitrary behavior sequences — may not hold against recommenders with anomaly-detection filters.

Relevance to Harnesses / Meta-Harnesses

CtrlBench-Rec is architecturally a meta-harness: it wraps an arbitrary black-box system in a structured pipeline that initializes, evolves, and deploys sub-agents, then aggregates their outputs into evaluation metrics — exactly the pattern of a harness that orchestrates other agents. The three-stage evolutionary loop (init → interact → fuse → re-interact) is a self-improving agent-cultivation harness, analogous to population-based meta-learning. The framework's reliance on the Oasis simulation platform as a lower-level harness illustrates a layered harness-of-harnesses architecture relevant to researchers studying how meta-harnesses compose. The "super probe" concept — where collective intelligence from many agents is distilled into a small elite set — directly instantiates the harness pattern of aggregating distributed exploration into a compact, reusable artifact.