Skip to content

FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments

🕒 Published (v1): 2026-04-28 02:21 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Failure-aware meta-agentic framework for open-source LLMs in tool-use [ACL 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FAMA is a two-stage meta-agentic framework that first profiles which failure modes dominate a baseline tool-use agent's trajectories, then dynamically routes execution through the minimal subset of specialized helper agents needed to suppress those failures. Applied to open-source LLMs (4B–72B) on multi-turn customer-service benchmarks, it achieves up to 27% absolute task-success improvement over static baselines.

Problem

Open-source LLMs deployed as tool-using agents in multi-turn conversational settings accumulate errors over long trajectories (domain-policy violations, hallucinations, parsing failures, premature stopping). Static scaffolding (e.g., always invoking all helper agents) does not help and can degrade performance, because different model backbones exhibit distinct dominant failure modes. Fine-tuning and RL are expensive and don't target observed failure patterns. Training-free, failure-aware orchestration for small open-source models is underexplored.

Method

Stage 1 – Failure profiling. A baseline agent (ReAct or Function Calling) runs on all tasks in domain \(T\). Failed trajectories \(\mathcal{F} = \{(\tau, \xi_\tau)\}\) are collected. For each failed trajectory, \(|E|=4\) independent error-analysis agents each evaluate the trajectory for one error category \(e \in E\) (Domain Policy Violation, Incorrect Retrieval from Complex Tool Outputs, Contextual Misinterpretation/Hallucination, Incomplete Fulfillment). Their outputs are concatenated and passed to an orchestrator agent that attributes the primary failure category \(\hat{E}_\tau\).

Stage 2 – Dynamic agent selection and execution. A mitigation agent, given \(\hat{E}_\tau\) and the functional definitions of all agents in the pool \(A\), selects the minimal subset \(A^* \subseteq A\) that addresses those failures. This subset—drawn from {Domain Constraints Extractor, Tool Suggestion Agent, Tool Output Reformulator, Planner, Decision Verifier, Memory Module}—injects targeted prior context into the tool-use agent's prompt before each decision step. The framework is summarized in Algorithm 1 and reuses only the selected agents per task, avoiding context-window saturation.

Key Contributions

  • Two-stage failure-aware orchestration: trajectory-level error attribution followed by minimal-agent context injection, without any parameter updates.
  • Demonstration that indiscriminate multi-agent context injection (IRMA) can hurt performance; selective routing strictly dominates.
  • Systematic error taxonomy (4 classes) with per-model failure-distribution analysis showing model-specific dominant failure modes.
  • Token-efficiency analysis showing FAMA outperforms thinking-augmented model variants while using fewer tokens and avoiding context-overflow failures.

Results

  • Ï„-bench Airline: FAMA outperforms ReAct / FC / IRMA by +4.63% / +11.57% / +5.27% (avg. over 5 runs); Qwen3-4B pass@1 improves from 32% (ReAct) to 37.6%.
  • Ï„-bench Retail: +5.30% / +8.96% / +6.15% over baselines; Qwen3-14B pass@1 jumps from 25.2% (ReAct) / 28.5% (IRMA) to 37.9%.
  • ACEBench: up to +27% over baselines.
  • Ï„-trait: up to +24% over baselines.
  • Gains hold across Qwen3-4B, Qwen3-14B, Qwen3-32B, Qwen2.5-72B-Instruct.
  • Qwen2.5-72B-Instruct as user simulator matches or exceeds GPT-4o pass@1 on Ï„-bench Airline (avg. ~47.6% vs. 45.2%).

Limitations

  • Orchestrator and mitigation agents use GPT-4o / GPT-4.1-mini as judges, introducing a proprietary dependency for the meta-level reasoning despite the framework targeting open-source deployment.
  • Failure profiling requires a prior run of the baseline agent on domain tasks; cold-start overhead is non-trivial for new domains.
  • The agent pool \(A\) is manually curated and domain-adapted; adding new error types or domains requires engineering new specialized agents.
  • Self-reflection variants of IRMA are excluded from comparison; integration with FAMA is left to future work.
  • Evaluation is limited to customer-service/tool-calling benchmarks; generalization to other agentic settings is untested.

Relevance to Harnesses / Meta-Harnesses

FAMA is a direct instance of a meta-harness: it does not act in the environment itself but operates one level above by observing agent trajectories, diagnosing failure modes, and dynamically composing the minimal execution harness for the underlying agent. The two-stage design—profile-then-route—is a reusable harness pattern applicable to any pool of specialized agents and any error taxonomy. For researchers building harnesses, FAMA concretely demonstrates that dynamic agent selection conditioned on observed failure distributions outperforms static "use all agents" orchestration, providing a principled design principle: harness composition should be failure-driven, not feature-driven. The framework's modular structure (error-analysis agents → orchestrator → mitigation agent → task agent) maps cleanly onto a general meta-harness architecture that can wrap arbitrary base agents.