FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments

🕒 Published (v1): 2026-04-28 02:21 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Failure-aware recovery framework for open-source LLMs in interactive tool environments

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FAMA is a training-free, two-stage meta-agentic framework that diagnoses the dominant failure modes of open-source LLMs in multi-turn tool-use tasks, then dynamically assembles a minimal subset of specialized helper agents to inject targeted context before each decision step. Evaluated on τ-bench, τ-trait, and ACEBench with Qwen models (4B–72B), it achieves up to 27% task-success improvement over strong baselines without any parameter updates.

Problem

Open-source LLMs (4B–72B parameters) used as tool-calling agents in multi-turn conversational benchmarks suffer from cascading decision errors—domain policy violations, hallucination, incorrect parsing of nested tool outputs, and premature task termination. Static prompting strategies and full multi-agent scaffolds (which inject all helper agents unconditionally) fail to address model-specific failure patterns and rapidly exhaust limited context windows. Training-based fixes (SFT, RL) are prohibitively expensive for long, partially observable trajectories with sparse rewards.

Method

FAMA operates in two stages.

Stage 1 — Failure Analysis. A baseline agent (ReAct or Function Calling) is run on the task set \(T\). Failed trajectories \(\xi_\tau\) are collected into a failure set \(F\). For each trajectory, \(|E| = 4\) independent error-analysis agents each examine \(\xi_\tau\) for one error category \(e \in E\): (1) Domain Policy Violation (DCV), (2) Incorrect Retrieval from Complex Tool Outputs (WRCO), (3) Contextual Misinterpretation and Hallucination (CM), (4) Incomplete Fulfillment or Early Stopping (IFU). Their textual verdicts and rationales are concatenated and fed to an orchestrator agent that attributes the primary failure mode \(\hat{E}_\tau\).

Stage 2 — Dynamic Agent Composition. Conditioned on \(\hat{E}_\tau\) and the functional definitions of the agent pool \(A\), a mitigation agent selects the minimal subset \(A^*_\tau \subseteq A\) needed to address the dominant error. The pool \(A\) contains: Domain Constraints Extractor (DCE), Tool Suggestion Agent (TSA), Tool Output Reformulator (TOR), Planner Agent, Decision Verifier Agent, and a Memory module. At inference, only the recommended agents are activated; their outputs are prepended as prior context to the tool-calling agent before its next decision step. This avoids context bloat from unconditional multi-agent injection (as in IRMA).

GPT-4o and GPT-4.1-mini serve as judgment agents in Stage 1 to ensure reliable error attribution, while the tool-calling backbone remains open-source throughout.

Key Contributions

• FAMA framework: training-free, failure-mode-driven dynamic agent selection for open-source LLMs in interactive tool use.
• Four-class error taxonomy for multi-turn tool-use failures, grounded in empirical trajectory analysis.
• Orchestrator + mitigation routing mechanism that selects the minimal effective helper-agent subset per task.
• Empirical demonstration that indiscriminate multi-agent injection (IRMA) can degrade performance, while targeted selection consistently improves it.
• Cross-benchmark generalizability shown on τ-bench (Airline, Retail), τ-trait (Telehealth, Telecom), and ACEBench (food delivery, telecom).

Results

• τ-bench Airline (Qwen3-4B): FAMA Pass@1 = 37.6% vs. ReAct 32.0%, FC 27.6%, IRMA 30.0%; gains of +4.6pp, +10.0pp, +7.6pp.
• τ-bench Retail (Qwen3-14B): FAMA Pass@1 = 37.9% vs. FC 32.6% and IRMA 28.5%.
• Overall τ-bench: FAMA outperforms ReAct by 4.63%, FC by 11.57%, IRMA by 5.27% averaged over five runs in Airline; and by 5.30%, 8.96%, 6.15% in Retail.
• ACEBench: up to 27% gain over baselines.
• τ-trait: up to 24% gain over baselines.
• FAMA with non-thinking Qwen models achieves stronger and more stable Pass@k curves than thinking/reasoning variants, which overflow context windows in multi-turn settings.
• Among open-source user simulators, Qwen2.5-72B-Instruct (Pass@1 ≈ 47.6%) most closely matches GPT-4o (Pass@1 ≈ 45.2%, with Qwen surpassing it on this metric).

Limitations

• Stage 1 failure attribution uses GPT-4o/GPT-4.1-mini as judge agents, introducing proprietary model dependency and cost despite the tool-calling backbone being open-source.
• Evaluated exclusively on customer-service simulation benchmarks (τ-bench, τ-trait, ACEBench); generalization to other agentic domains (code, science, web navigation) is unverified.
• The agent pool \(A\) and error category set \(E\) are predefined; adding new failure modes or domains requires manual extension of the taxonomy and pool.
• Integration with the Self-Reflection variant of IRMA is left for future work, leaving a potentially complementary direction unexplored.
• Reasoning/thinking model variants are excluded from the primary evaluation because their internal token consumption causes context overflow—a practical constraint that limits the study's scope.

Relevance to Agentic AI / LLM Agents

FAMA directly addresses a central challenge in deploying LLM agents under real-world resource constraints: error accumulation over long tool-use trajectories in small open-source models. Its core insight—that static scaffolding is insufficient because failure modes are model- and domain-specific, and that dynamically routing to the minimal mitigating agent subset outperforms unconditional multi-agent injection—is a broadly applicable design principle. The work extends the trajectory of failure-aware orchestration (beyond reflection and episodic hints) into model-specific, inference-time composition, which is directly relevant to anyone building reliable agentic pipelines with constrained LLMs. It also contributes a reusable four-class error taxonomy and an empirical methodology for surfacing dominant failure modes from baseline runs.