AutoQual: An LLM Agent for Automated Discovery of Interpretable Features for Review Quality Assessment¶

🕒 Published (v1): 2025-10-09 11:11 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AutoQual is an autonomous LLM agent framework that converts tacit domain knowledge (embedded in labeled text data) into explicit, interpretable feature functions for quality scoring tasks. It orchestrates hypothesis generation, autonomous tool implementation, and reflective beam-search feature selection guided by a dual-level memory system. Deployed on a billion-user e-commerce platform, it yields +0.79% average reviews viewed per user and +0.27% reader conversion rate in A/B testing.

Problem¶

Hand-crafted features for review quality assessment are domain-specific and brittle; deep learning models are black-boxes that optimize for semantics rather than quality. No prior framework automatically discovers a compact, interpretable, and maximally predictive feature set from unlabeled quality signals, especially across domains with evolving content patterns.

Method¶

AutoQual operates in three sequential phases, looped iteratively:

Hypothesis Generation — the LLM instantiates multiple expert personas (multi-perspective ideation) and performs contrastive analysis on sampled high/low-quality examples to produce a deduplicated candidate feature pool $S_{\text{cand}}$.
Autonomous Tool Implementation — for each feature hypothesis $f \in S_{\text{cand}}$, the agent decides between a Python code tool or an LLM prompt tool, then iteratively generates, validates on a small sample, and refines until acceptable. The finalized tool annotates the full dataset, yielding feature values.
Reflective Feature Search — beam search (width $m=5$) greedily selects features maximizing conditional mutual information: $$f^* = \arg\max_{f_{\text{new}} \in S_{\text{cand}} \setminus S_{\text{current}}} I(Y;\, f_{\text{new}} \mid F_{S_{\text{current}}})$$ After each selection of $k=10$ features, an intra-task reflection step analyzes MI scores, distills principles of feature effectiveness, and proposes new hypotheses that are added to $S_{\text{cand}}$ before re-running beam search. This cycle repeats for a fixed number of iterations.

A dual-level memory supports the loop: - Intra-task memory (working memory): maintains tested features and MI scores within one run, enabling within-session reflection. - Cross-task memory (long-term): after task completion, synthesizes the feature set and MI into a persistent knowledge base; queried at hypothesis generation for new tasks to bootstrap the candidate pool.

Key Contributions¶

First framework for automated interpretable quality feature discovery in online reviews.
Dual-level memory enabling both within-task adaptation and cross-task knowledge transfer.
Propose-validate-refine cycle for autonomous tool (code or prompt) generation, avoiding manual feature operationalization.
Large-scale industrial validation on a billion-user platform with live A/B test results.
Demonstrated generalizability beyond reviews: text persuasiveness, automated essay scoring, toxicity detection.

Results¶

Meituan (CTR): AutoQual $r_s = 0.5661$, AutoQual+PLM $r_s = 0.5833$ vs. Fine-tuned PLM $r_s = 0.5656$ (best prior).
Cell Phones: AutoQual $r_s = 0.6992$, AutoQual+PLM $r_s = 0.7105$ vs. Fine-tuned PLM $r_s = 0.6977$.
Office Products: AutoQual $r_s = 0.6646$, AutoQual+PLM $r_s = 0.7189$ vs. Fine-tuned PLM $r_s = 0.6934$.
Clothing: AutoQual $r_s = 0.6278$, AutoQual+PLM $r_s = 0.8014$ vs. Fine-tuned PLM $r_s = 0.7816$.
Grocery: AutoQual $r_s = 0.7194$, AutoQual+PLM $r_s = 0.7920$ vs. Fine-tuned PLM $r_s = 0.6770$.
AutoQual+PLM achieves best $r_s$ on all five datasets; task-specific baselines (TNN, SEHP, BHeLP-CoRT) are consistently outperformed.
Ablation: Removing contrastive analysis: avg $r_s$ drop of $0.0537$; removing multi-perspective ideation: $-0.0335$; removing intra-task memory: $-0.0170$.
Cross-task memory (without intra-task) reduces agent LLM token consumption by 44.95% and annotation LLM consumption by 29.79% while matching full AutoQual performance.
Generalization: Persuasiveness $r_s = 0.432$ (vs. fine-tuned PLM $0.435$, manual $0.447$); toxicity AutoQual+PLM F1 $0.8364$ / AUROC $0.9203$ vs. fine-tuned PLM $0.8224$ / $0.9078$.
A/B test (industrial): +0.79% reviews viewed per user, +0.27% reader conversion rate.

Limitations¶

Hypothesis generation, tool implementation, and reflective loops all call a large LLM (DeepSeek-V3.2-Exp in thinking mode), making per-task compute cost substantial; cross-task memory is the only mechanism to amortize this.
Tool quality relies on LLM self-validation, which can silently pass low-quality tools when the LLM misjudges its own output.
Beam search with fixed $k$ and $m$ may miss globally optimal feature sets; mutual information estimation via KNN can be noisy for small or heavily skewed datasets.
Cross-task memory retrieval is not described in detail (no embedding/retrieval ablation), leaving its robustness on distant domains uncertain.
Main experiments deliberately exclude cross-task memory; real-world gains from accumulated memory are only partially characterized.
Private Meituan dataset is not publicly released, limiting full reproducibility.

Relevance to Harnesses / Meta-Harnesses¶

AutoQual is a textbook meta-harness: it wraps multiple LLM sub-agents (persona ideation, contrastive analyzer, tool generator, validator, reflector) inside a deterministic outer loop (beam search + iteration count) with a persistent state layer (dual-level memory). The propose-validate-refine inner loop for tool construction is itself a micro-harness pattern—structurally identical to the self-debugging harnesses used in code generation research. The cross-task memory that bootstraps future runs and reduces token costs by ~45% directly demonstrates how persistent state in a meta-harness can compound efficiency gains across invocations, a key design concern for researchers building reusable orchestration scaffolds.