From Accuracy to Impact: The Impact-Driven AI Framework (IDAIF) for Aligning Engineering Architecture with Theory of Change

🕒 Published (v1): 2025-12-09 10:21 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IDAIF is a six-layer AI architectural framework that maps Theory of Change's five-stage causal model (Inputs→Activities→Outputs→Outcomes→Impact) onto corresponding AI system layers (Data→Pipeline→Inference→Agentic→Normative), plus a cross-cutting Assurance Layer. The framework reframes AI design from metric optimization toward explicit sociotechnical impact, formalizing each layer with mathematical specifications. Three case studies in healthcare, cybersecurity, and software engineering demonstrate the backward-mapping process from impact goal to technical implementation.

Problem

Current AI systems optimize technical performance metrics (accuracy, latency, benchmarks) while treating ethical and societal alignment as post-hoc constraints. This produces models that satisfy proxy objectives while generating negative externalities—demographic bias, hallucinations, reward hacking—especially under distributional shift. No existing framework (VSD, MLOps, RLHF, Constitutional AI) provides an integrated end-to-end architecture from raw data governance to societal impact specification; each addresses isolated concerns without a unifying causal structure.

Method

IDAIF establishes a bidirectional five-layer stack plus Assurance Layer, each grounded in a formal technique:

• Normative Layer (Impact): Multi-objective Pareto optimization using Minimax Pareto Fairness (MMPF)—minimizes worst-case group-conditional risk via adaptive reweighting λᵢ ∝ exp(η·Rᵢ(θ)).
• Agentic Layer (Outcomes): Hierarchical multi-agent orchestration (Planner/Executor/Critic roles) with dynamic scope assignment S(t) = ⌊3·(1−Rₜ)·C·V⌋ drawn from the AWS Agentic Security Scoping Matrix.
• Inference Layer (Outputs): Causal-DAG Construction and Reasoning (CDCR)—LLM first constructs a DAG G_q = arg max P(G|q,K_q) constrained to knowledge graph subgraphs, then generates responses along valid causal paths to mitigate hallucination.
• Pipeline Layer (Activities): Adversarial debiasing (minimax predictor/adversary on sensitive attributes) combined with PPO-based RLHF with KL-divergence penalty β·D_KL(πθ||π_ref).
• Data Layer (Inputs): Fairness-Aware Demonstration Selection (FADS) for in-context learning; demographic balance constraints |P_data(A=a) − P_target(A=a)| ≤ δ.
• Assurance Layer (cross-cutting): Guardian architecture with three tiers—fast AI model, human-in-the-loop escalation, and guardian models (fact-checkers, toxicity filters, consistency checkers) monitoring IID, completeness, and correlation-causation assumption failures.

Design follows backward mapping: specify Impact first, derive each layer's requirements downward to Inputs.

Key Contributions

• Formal bijective mapping between ToC's five stages and AI architectural layers, providing a translation layer from abstract values to concrete engineering specifications.
• Mathematical formulations for each layer: MMPF (Pareto fairness), SCM/Causal-DAG inference, adversarial debiasing loss, PPO-RLHF objective, FADS demonstration selection.
• Assurance Layer with guardian architecture classifying and handling three assumption-failure modes (IID drift, distributional incompleteness, spurious correlation).
• Three domain-validated case studies demonstrating backward ToC mapping and expected quantitative outcomes.

Results

All results are stated as expected/evaluation outcomes from case study deployments (not held-out benchmark comparisons against published baselines):

• Healthcare (CDSS fairness): Overall AUC = 0.86; Equalized Odds Difference = 0.094 (target <0.10); 47% reduction in demographic disparity vs. single-task baseline.
• Cybersecurity (SOC agent): 73% reduction in analyst alert volume; MTTR 4.2 h → 1.1 h; zero false-positive automated containment actions over 6 months; 89% of routine containment automated under Scope 1–2.
• Software engineering (code gen): Security vulnerability rate 40% → 3.2% (92% reduction); architectural conformance 61% → 94%; developer acceptance 23% → 67%; refactoring time 45 min → 8 min per component.

No comparison to external published baselines is provided; all figures are internal before/after or threshold-relative.

Limitations

• Case study results are "expected" outcomes, not independently replicated experiments; no ablation studies isolate individual layer contributions.
• The framework is descriptive-prescriptive: it assembles existing techniques (RLHF, adversarial debiasing, MMPF, CDCR) without novel algorithmic development or proof of compositional correctness.
• No empirical validation that the ToC causal chain actually holds in deployment; assumption monitoring is architected but not evaluated for recall/precision on assumption failures.
• Backward mapping from Impact to Data requires stakeholder consensus on value priorities; the paper provides no methodology for resolving genuine value conflicts beyond Pareto front navigation.
• Scope assignment formula (Equation 3) and Policy Guardian constraints (Equation 12) rely on quantities (risk, confidence, reversibility) whose reliable estimation is itself an open problem.

Relevance to Harnesses / Meta-Harnesses

IDAIF is architecturally a meta-harness: it does not implement any single capability but orchestrates multiple AI subsystems (planner agents, executor agents, critic/guardian agents, RAG pipelines, RLHF trainers) under a unified causal accountability structure. The Assurance Layer functions as a runtime harness that wraps all other layers, routing outputs through guardian models and human checkpoints—analogous to a meta-harness that intercepts inter-component communication for validation. The hierarchical Agentic Layer with dynamic scope assignment is a concrete multi-agent orchestration pattern relevant to anyone building harnesses that must manage autonomy levels and escalation paths. The backward ToC mapping methodology—deriving pipeline, data, and orchestration requirements from a high-level impact specification—provides a design discipline directly applicable to meta-harness authoring, where the harness itself must encode the intent, not just the mechanics, of the workflow it governs.