Engineering Trustworthy Agentic AI for Critical Systems¶

🕒 Published (v1): 2026-07-20 22:28 UTC · Source: Arxiv · link

Why this paper was selected

Engineering framework for trustworthy agentic AI in critical systems; evaluation and safety criteria

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This survey argues that trustworthiness—not capability—must be the primary engineering property of agentic AI deployed in critical infrastructure. It proposes a five-dimension trustworthiness model mapped onto a ten-stage agentic workflow and applies it across four constraint-bound domains: power systems, autonomous vehicles/robotics/UAVs, HPC, and communication networks. The central insight is that the trustworthiness gaps across these domains are structurally identical, pointing toward a single reusable cross-domain assurance framework.

Problem¶

Existing agentic AI surveys focus on reasoning capability, coordination mechanisms, and performance metrics, while surveys on trustworthy AI address static or single-decision models. No existing framework treats trustworthiness as a first-class, workflow-level engineering property for autonomous, sequential, tool-integrated agentic systems operating in critical infrastructure where failures cause cascading physical, economic, or safety consequences.

Method¶

The paper develops a trust-centered analytical structure with three components:

Unified agentic taxonomy: classifies architectures into reasoning-acting interleaving (ReAct), self-reflective/memory-augmented (Reflexion), tool-augmented, deliberative planning (Tree-of-Thoughts), and multi-agent coordination systems.
Five trustworthiness dimensions cross-cutting the taxonomy:
Safety and constraint satisfaction (operational envelopes, SOTIF)
Robustness and reliability (ODD definition, distribution shift, failure-mode escalation)
Transparency and interpretability (traceable goals, tool calls, memory updates—not just output explanations)
Accountability and auditability (tamper-evident logs, versioned policies, traceable tool use)
Privacy and security (prompt injection, tool hijacking, least-privilege sandboxing)
Ten-stage agentic assurance workflow: expands the generic observe–reason–act–update loop into: perception/observation → reasoning → planning → tool use → communication/coordination → memory update → validation → action/execution → monitoring/re-evaluation → audit/logging. Each stage is interrogated for (i) relevant threat/failure model, (ii) embedded prevention/detection/correction mechanisms, and (iii) quantitative evidence reported.

This framework is then instantiated across four engineering domains, comparing simulators, benchmarks, deployment metrics, and domain-specific failure modes.

Key Contributions¶

Unified taxonomy of agentic LLM architectures organized for trust analysis, not capability comparison
Five-dimension trustworthiness model grounded in functional safety standards (ISO 21448/SOTIF), cybersecurity guidance, and EU AI Act requirements
Ten-point agentic assurance workflow enabling per-stage trust gap identification
Consolidated set of quantitative metrics spanning tool use, memory coherence, planning quality, policy adherence, and goal drift—reportable independently of task success
Cross-domain analysis of four critical infrastructure domains revealing that four apparently distinct engineering problems share a common gap structure
Path toward a reusable cross-domain assurance framework analogous to aviation/automotive graded certification regimes (DO-178C, ISO 26262)

Results¶

The paper is a survey/position work; it does not report empirical benchmark numbers. Key qualitative findings reported in the text:

All four surveyed domains exhibit the same four recurring gaps: (1) absence of domain-aware safety guarantees under distribution shift and adversarial conditions; (2) lack of standardized trust-oriented evaluation; (3) limited integration of formal verification and runtime monitoring; (4) inadequate human-agent oversight
Current agentic architectures share a common pattern: they improve capability and partial observability but lack compositional safeguards and evidentiary coverage across all five trustworthiness dimensions
ReAct-style agents inherit safety from the tool environment rather than enforcing it internally; self-evaluator-based refinement (ReST meets ReAct) risks reinforcing incorrect LLM judgments
Multi-agent systems introduce emergent inter-agent trust risks (error propagation through agent networks) beyond intra-agent reliability concerns

Limitations¶

No original empirical evaluation; all claims are synthesized from surveyed literature, so gap characterizations reflect the surveyed corpus rather than independent measurement
The four selected engineering domains do not cover all critical infrastructure (e.g., healthcare, finance, aviation control are not primary case studies)
The proposed cross-domain assurance framework is outlined as a direction, not a fully specified artifact—no formal specification, toolchain, or certification prototype is provided
Quantitative metrics catalog consolidates existing proposals but does not validate their discriminative power or deployment feasibility
The paper acknowledges but does not resolve the tension between LLM opacity and the transparency/auditability requirements it mandates

Relevance to Agentic AI / LLM Agents¶

This survey directly addresses the operational deployment frontier for agentic LLM systems, reframing the canonical research question from "can agents perform task X?" to "can agents be certified safe to perform task X?" The five-dimension / ten-stage framework provides a vocabulary and structure for evaluating any agentic architecture—ReAct, Reflexion, MAS orchestration—against trust requirements rather than benchmark scores alone. The insight that trustworthiness gaps are structurally invariant across domains motivates building shared assurance infrastructure (runtime monitors, audit logs, formal constraint layers) rather than domain-specific patches. For researchers tracking agentic AI, this paper is a useful forcing function for identifying where current agent designs (tool use, memory, multi-agent coordination) leave trust obligations unmet.