Skip to content

TAI3: Testing Agent Integrity in Interpreting User Intent

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TAI3 is an API-centric stress-testing framework that systematically discovers intent integrity violations in LLM agents—cases where the agent misinterprets or incorrectly executes a user's natural-language instruction. It combines equivalence-class-inspired semantic partitioning of API parameter spaces with intent-preserving mutation and a strategy memory to efficiently surface agent errors across diverse toolkits.

Problem

LLM agents translating natural language into API calls frequently misinterpret user intent in subtle ways (e.g., granting permanent instead of recurring access, hallucinating invalid dates, or silently filling in under-specified parameters). Existing safety benchmarks use fixed test cases that miss evolving toolkit behaviors, while classical software testing assumes structured inputs and cannot handle natural language ambiguity. There is no rigorous, measurable method for stress-testing whether agents faithfully preserve user intent across the full semantic space of their APIs.

Method

TAI3 operates in two stages:

Stage 1 — Semantic Partitioning: For each API parameter, TAI3 uses an LLM to partition the value domain into equivalence classes under three intent categories: VALID (well-formed, executable intent), INVALID (out-of-scope or malformed), and UNDERSPEC (missing required information). Each (parameter, category, equivalence class) triple forms a "cell," and one realistic seed task is generated per cell to guarantee coverage of the semantic input space.

Stage 2 — Intent-Preserving Mutation: Starting from a seed task, a mutator LLM generates paraphrased variants. An LLM-based intent consistency checker filters out any variant that alters the original intent I(u). Remaining candidates are ranked by a small language model (SLM, phi4-mini) using a log-likelihood score measuring how well the SLM can reconstruct the original intent from the mutated task—lower score = higher ambiguity = higher error likelihood. The top-ranked tasks are executed against the target agent, and execution trajectories are judged for violations.

Strategy Memory: Successful mutation patterns are stored indexed by parameter datatype and intent category. For new seed tasks, the top-3 retrieved strategies guide mutation, enabling cross-API transfer of effective perturbation patterns.

Key Contributions

  • TAI3 framework: first API-centric, coverage-guided stress-testing framework targeting LLM agent intent integrity (distinct from jailbreaking or policy-violation testing).
  • Semantic partitioning: equivalence-class-based decomposition of natural-language API input space into measurable VALID/INVALID/UNDERSPEC cells.
  • Error-likelihood estimator: SLM-based ranking of mutated tasks (Eq. 1) that reduces costly agent queries while prioritizing high-risk inputs.
  • Evergreen strategy memory: datatype- and category-aware retrieval and reranking of successful mutation patterns for cross-task adaptation.
  • EESR / AQFF metrics: new evaluation metrics for agent stress-testing (error-exposing success rate and average queries to first failure).
  • Coverage gap analysis: empirical demonstration that AgentSafetyBench and ToolEmu leave large fractions of semantic partitions uncovered.

Results

  • Evaluated on 80 toolkit APIs, 233 parameters across 5 domains (finance, healthcare, smart home, logistics, office), using ToolEmu data.
  • TAI3 consistently outperforms the SelfRef baseline (iterative self-reflection) in EESR across all domains and intent categories.
  • Finance / VALID: +19.5 pp over SelfRef (GPT-4o-mini target); +15.5 pp (Llama-3.1-8B target).
  • Office / UNDERSPEC: +10.0 pp (GPT-4o-mini target); +16.3 pp (Llama-3.1-8B target).
  • TAI3 reduces AQFF by up to 12% in UNDERSPEC settings vs. SelfRef.
  • SLM-based ranking consistently outperforms random selection and "select-last-5" self-reflection across all values of k.
  • Generalizes to stronger closed-source targets: Top-5 EESR with GPT-4o-mini tester reaches 85.7% against DeepSeek-R1-70B.
  • Open-source targets (Llama-3.3-70B, DeepSeek-R1-70B) show higher EESR than closed-source ones (Claude-3.5-Haiku, Gemini-2.5-Pro, GPT-o3-mini), indicating greater vulnerability.
  • Weaker testing models (Llama-3.1-8B) successfully expose errors in stronger target models, confirming cost-efficient generalization.

Limitations

  • Requires access to the agent's full API-calling trajectory (tool call traces); inapplicable to commercial agents that only expose high-level text outputs.
  • The parameter-partition form is constructed semi-automatically but still requires expert-defined intent category boundaries; scalability to very large or undocumented APIs is unclear.
  • Evaluation is limited to agents that invoke APIs from documented toolkits; agentic tasks involving web browsing or open-ended tool use are not covered.
  • The SLM error-likelihood estimator is a proxy (log-likelihood of intent reconstruction) and may not perfectly predict agent failures on novel datatypes or multi-hop reasoning chains.

Relevance to Agentic AI / LLM Agents

TAI3 addresses a critical but underexplored failure mode of deployed LLM agents: not adversarial attacks, but ordinary benign tasks where the agent silently diverges from user intent due to natural-language ambiguity or evolving APIs. For researchers tracking agentic AI, this framework provides a rigorous, coverage-measurable complement to benchmark-based evaluations and connects directly to reliability, safety, and tool-use robustness lines of work. The finding that even weaker SLMs can expose failures in frontier models (Claude-3.5-Haiku, Gemini-2.5-Pro) is practically significant for red-teaming pipelines. The semantic partitioning methodology also generalizes beyond testing—it offers a principled vocabulary for characterizing the intent space of any API-calling agent.