Skip to content

Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition

🕒 Published (v1): 2025-04-26 00:55 UTC · Source: Arxiv · Venue: ICML 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MATCHA is a six-agent pipeline for conversational game recommendation that decomposes the task across specialized agents for intent parsing, tool-augmented retrieval, multi-LLM ranking, reflection, explanation, and adversarial risk control. Evaluated on the Roblox OMuleT dataset, it outperforms six baselines on eight metrics, gaining +20% Hit@5 and blocking 97.9% of adversarial prompts. The decomposition explicitly targets three game-specific failure modes: complex interactive constraints, knowledge recency gaps, and safety/transparency risks.

Problem

Existing CRS work focuses on static-catalog domains (movies, books) where LLM pretraining data is rich and content is passively consumed. Games present three compounding challenges absent in those domains: (1) preference depends on interactive factors (mechanics, skill level, hardware, social mode), not just genre/theme; (2) game catalogs change rapidly via user-generated content and are underrepresented in LLM pretraining corpora, causing accuracy degradation on niche/new titles; (3) open-ended game-platform conversations increase exposure to adversarial jailbreak prompts (e.g., harm-seeking queries disguised as game requests), which existing CRS systems almost entirely ignore.

Method

MATCHA is formally defined as \(M = \{A_1, \ldots, A_n\}\) where each \(A_i : Q \rightarrow X_i\) maps user query \(q\) to an intermediate output (intent vector, candidate list, explanation draft). The pipeline optimizes:

\[\max_{\{r_i\},\{e_i\}} \sum_{i=1}^k \text{Rel}(r_i, q) + \alpha \cdot \text{Qual}(e_i, r_i, q)\]

Risk Control Module (pre-filter): A Jailbreak Prevention Agent applies three methods in sequence—(i) RA-LLM random token-drop to detect sensitivity to perturbation, (ii) chain-of-thought-based intent detection via an auxiliary LLM, and (iii) policy-defined fallback behaviors. A secondary Dangerous Content Detection Agent filters both input queries and output recommendations, each returning a binary label.

Candidate Generation: An intent agent extracts structured preferences; a Tool-Augmented Candidate Generation Agent queries 10+ specialized APIs (real-time game databases, genre filters, trend analyzers, ratings/review systems, platform compatibility APIs) to build a diverse candidate pool.

Ranking Agent: Two LLMs (GPT-4o + Gemini) score candidates independently across five dimensions (popularity, user-preference match, historical similarity, genre alignment, age suitability); scores are combined via weighted average. An exploratory hyperparameter allows deliberate out-of-preference suggestions.

Reflection Agent: A self-reflection pass re-ranks the top-\(k\) candidates using full "game profiles" (detailed metadata), restricted to a subset of top candidates for cost efficiency.

Explanation Agent: Generates per-recommendation rationales across four dimensions—category preferences, similarity to past titles, demographics, and popularity/novelty—assembled via structured LLM prompts and aggregated into a summary.

Key Contributions

  • Novel multi-agent architecture with six specialized agents covering the full CRS pipeline for the game domain
  • First CRS system to explicitly integrate adversarial jailbreak defense as a core module, combining RA-LLM random-drop, CoT intent detection, and policy fallbacks
  • Multi-LLM ranking with weighted ensemble (GPT-4o + Gemini) to overcome static single-model knowledge limits
  • Evaluation protocol spanning eight metrics (factuality, Hit@k, Precision@k, Pop50, RPop50, Entropy, MaxFreq, Explanation Score) on real Roblox user requests plus jailbreak benchmarks (WildJailbreak, DAN)
  • Practical deployment insights for production multi-agent CRS on Roblox

Results

  • Hit@5: MATCHA .29 vs. best baseline OMuleT .24 (+20.8%); Hit@10: .39 vs. .33 (+18.2%)
  • Popularity bias (RPop50@5): MATCHA 2.05 vs. MAgent 3.83 (–46.5%); paper reports –24% reduction in popularity bias overall
  • Factuality@5: .99, matching OMuleT and surpassing MACRec (.92), MACRS (.85)
  • Coverage (Entropy@5): MATCHA 8.40; MaxFreq@5: .09 (lowest repetition among GPT-4o methods)
  • Jailbreak Prevention: 97.9% defense rate; all baselines except LLaMA MAgent fail (✘)
  • Explanation Score: 4.2/5 (virtual judge), 3.97/5 (human domain experts)—close agreement
  • Ablation: removing Reflection Agent reduces relevance; removing multi-LLM collaboration reduces ranking accuracy; removing tools significantly lowers coverage and novelty

Limitations

  • Reflection Agent is restricted to a candidate subset for cost efficiency, reducing diversity slightly compared to full-set reflection
  • Game profiles used in reflection are long; including them in earlier stages is computationally prohibitive, creating an accuracy/efficiency trade-off
  • OMuleT outperforms MATCHA on Entropy@10 (9.21 vs. 8.65), meaning coverage diversity is not Pareto-dominant
  • The jailbreak defense rate (97.9%) implies ~2% adversarial pass-through in a safety-critical setting
  • Generalization to non-Roblox game platforms is not directly verified; the Roblox-specific catalog and APIs may not transfer without re-engineering the tool suite
  • Human evaluation (Appendix F) was conducted by domain experts, not end users; real-world alignment may differ

Relevance to Harnesses / Meta-Harnesses

MATCHA is a textbook application of the modular multi-agent harness pattern: a fixed pipeline of specialized agents where each stage produces a typed intermediate output consumed by the next, with explicit routing logic (binary labels from safety agents determine whether execution continues). The Risk Control Module functions as a pre- and post-execution guard harness—a pattern directly analogous to pre/post-hook design in meta-harnesses. The multi-LLM Ranking Agent is an ensemble sub-harness that aggregates outputs from heterogeneous models via weighted voting, a coordination primitive relevant to any harness that must reconcile multiple LLM opinions. The ablation study provides rare empirical evidence that each harness component contributes independently, validating modular decomposition as an engineering principle rather than a design choice made for presentation.