Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance¶

🕒 Published (v1): 2025-07-23 02:12 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ARIA (Adaptive Reflective Interactive Agent) is an LLM agent framework that enables continuous learning at test time by using structured self-dialogue to detect knowledge gaps, proactively soliciting targeted human-expert guidance within a query budget $B$, and integrating received knowledge into a timestamped, conflict-aware internal repository. It is deployed on TikTok Pay serving 150M monthly active users for customer due diligence (CDD) name screening.

Problem¶

LLM agents cannot adapt to rapidly changing domain-specific rules or knowledge distributions after deployment. Offline fine-tuning requires labeled data and retraining cycles; RAG and ICL provide static snapshots; autonomous self-improving agents lack a mechanism to receive and integrate authoritative corrections. In regulated, evolving domains (e.g., sanctions screening), this causes unreliable or non-compliant decisions.

Method¶

ARIA processes a sequential stream of instances $X = (x_1, \ldots, x_N)$, maintaining internal state $\Theta_i \approx KR_i$ as a structured Knowledge Repository. Two core modules govern adaptation:

Intelligent Guidance Solicitation (IGS). For each instance, the agent generates an initial prediction and then answers $N_{RQ}$ fixed reflective questions (e.g., "What assumptions did you make?", "Do you have the domain knowledge required?") via self-dialogue $D_i^{\text{self}}$. From this, it derives $\text{conf}_i \in \{\text{High, Moderate, Low}\}$. If $\text{conf}_i \in \{\text{Moderate, Low}\}$ and remaining budget allows, it formulates a typed query $q_i \in Q$ (label request, rule clarification, explanation) and sends it to a human-expert oracle $O$.

Human-Guided Knowledge Adaptation (HGKA). Received feedback $h_i$ is parsed into knowledge assertions. Each new item $K_{\text{extracted}}$ is compared to existing repository items via semantic similarity $\text{Sim}(\cdot, \cdot) > \tau_{\text{sim}}$; an LLM then determines whether the new item supersedes, conflicts with, or is consistent with each related item, updating item status $S \in \{\text{Valid, PotentiallyOutdated, Superseded}\}$ and timestamps $ts_{\text{validated}}$ accordingly. Unresolvable conflicts trigger active clarification queries back to the expert.

Temporally-informed retrieval scores repository items as: $$\text{Score}(k, x_j, t) = W_S(k.S) \times \exp(-\lambda(t - k.ts_{\text{validated}})) \times S_R(k, x_j)$$ so that stale or superseded knowledge is down-weighted in context construction.

The overall objective is: $$\max_{\pi, f, \text{query\_strategy}} \sum_{i=1}^N \text{Eval}(\hat{y}_i, y_i^*) \quad \text{s.t.} \sum_j c(q_j) \leq B$$

Key Contributions¶

ARIA framework: general test-time learning loop combining structured self-reflection, HITL querying, and a managed knowledge base.
IGS module: replaces heuristic confidence thresholds with multi-question self-dialogue to trigger and shape expert queries.
HGKA module: timestamped knowledge repository with explicit validity states, LLM-based conflict detection, and active clarification generation.
Temporally-informed retrieval: composite scoring that jointly penalizes stale status, recency decay, and low semantic relevance.
Industrial deployment: evaluated on 11,846 real CDD cases from TikTok Pay; confirmed in production at scale.

Results¶

TikTok Pay CDD (Table 1), GPT-4o: ARIA at $B=1000$ reaches 0.8910 Sensitivity / 0.8026 Specificity, vs. best active-learning baseline (Simple Uncertainty Sampling) at 0.8718 / 0.7853; static GPT-4o baseline: 0.7756 / 0.6864.
Efficiency (Table 3): Average Handling Time per case drops from ~12 min (human experts) to 0.13–0.41 min across ARIA budget variants.
Ablation (Table 2, $B=100$, GPT-4o): removing Self-Dialogue drops Sensitivity from 0.8333 → 0.8141; removing KR Conflict Resolution → 0.8012; removing Temporally-Informed KR → 0.8333/0.7341 (Specificity drop); Labels-Only queries → 0.7949/0.7139.
ARIA with Qwen2.5-7B outperforms GPT-4o static agent on Sensitivity at $B \geq 100$, demonstrating that the adaptation framework compensates for a weaker base model.

Limitations¶

Human expert dependency at inference time: the framework requires real-time expert availability; throughput is bounded by oracle response latency.
Budget sensitivity: performance degrades significantly at very low budgets ($B=50$) and converges toward a ceiling well below full oracle access ($B=3121$ gives 0.9428/0.8814).
Domain scope: primary evaluation is binary classification (Match/Non-Match); generalization to open-ended generation tasks is asserted but not deeply benchmarked.
Conflict resolution accuracy: relies on LLM semantic comparison, which may silently fail on subtle rule contradictions or near-duplicate paraphrases.
No forgetting mechanism: the repository grows monotonically; scalability of retrieval at very large $|KR|$ is unaddressed.

Relevance to Harnesses / Meta-Harnesses¶

ARIA is structurally a meta-harness: it wraps a base LLM with an orchestration layer that manages knowledge state, routes decisions between self-execution and human escalation, enforces a budget policy, and maintains a mutable intermediate store between inference steps—precisely the concerns of harness design. The IGS → HGKA loop formalizes a pattern relevant to any harness that must decide when to invoke an external oracle and how to durably integrate its output, making it a concrete reference design for test-time adaptation components in broader agent pipelines. The explicit budget constraint $B$ and the composite retrieval score also provide quantitative handles for tuning harness behavior that meta-harness designers can directly parameterize.