The Hitchhiker's Guide to Agentic AI: From Foundations to Systems¶

🕒 Published (v1): 2026-06-22 17:48 UTC · Source: Arxiv · link

Why this paper was selected

Practitioner-oriented full-stack agentic AI reference; foundations to production, covers harness architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

A comprehensive textbook survey (v1.2.2, 200+ pages) covering the full stack of agentic AI: LLM architecture and optimization, RL-for-LLMs (PPO/DPO/GRPO and their variants), and distributed training infrastructure. It synthesizes practitioner-level detail across foundations and systems, functioning as a unified reference for building production-grade agentic pipelines.

Problem¶

The agentic AI landscape lacks a single cohesive reference that connects low-level LLM mechanics (tokenization, Flash Attention, speculative decoding) through RL training algorithms (RLHF, GRPO variants) to distributed system design (parallelism, fault tolerance, cost modeling). Fragmented literature forces practitioners to stitch together disparate papers and blog posts.

Method¶

Structured as a two-part textbook. Part I (Foundations) builds from transformer internals (BPE tokenization → multi-head attention → MoE → LoRA), through decoding strategies (greedy/beam/nucleus/speculative), to prompt engineering and hardware-aware inference (PagedAttention/vLLM, Flash Attention 1–4, GPU roofline model). Part II (RL Methods for LLMs) formalizes text generation as an MDP and covers the full RLHF pipeline, deriving PPO (clipped surrogate objective), DPO (preference log-ratio loss), and GRPO (group-relative baseline), with implementation recipes for TRL. A dedicated chapter taxonomizes 12+ GRPO variants (DAPO, GSPO, Dr. GRPO, 2-GRPO, SAPO, VESPO, DPPO, ScaleRL/CISPO, GDPO, GOPO). System chapters cover 3D parallelism (TP×PP×DP/FSDP), decoupled inference/training architectures, weight sync strategies, fault tolerance, and cost estimation.

Key Contributions¶

Single-reference synthesis from raw transformer math through distributed RLHF system design
Most comprehensive taxonomy of GRPO variants in one place (12+ algorithms with motivation and comparison)
Unified derivation of PPO, DPO, GRPO, KTO, IPO, ORPO, SimPO with TRL implementation pointers
Reward model chapter covering Bradley-Terry, PRMs vs ORMs, rule-based rewards (RLVR), multi-objective and listwise rank-based rewards
System chapter with quantitative analysis of the "4-model memory challenge" in RLHF, decoupled architecture design, MFU measurement, and cloud cost estimation

Results¶

No original empirical benchmarks — this is a survey/textbook. Individual sections cite established results from the literature (e.g., PagedAttention memory savings for 70B models, scaling laws for Best-of-N, MFU targets) but no new numbers are reported by the authors.

Limitations¶

Text is heavily truncated in the provided excerpt; agentic system chapters (Part III and beyond, if any) are not visible
No original experiments; all quantitative claims are inherited from cited works and cannot be independently verified from this text
Version 1.2.2 status suggests ongoing revision; coverage may be uneven across chapters
Depth of agentic orchestration (multi-agent coordination, tool use, memory) is not visible from the provided TOC

Relevance to Harnesses / Meta-Harnesses¶

This guide is directly foundational for harness builders: the RLHF pipeline description (PPO rollout buffer → optimization loop → weight sync) maps directly onto the lifecycle of a training harness, while the system architecture chapter (decoupled inference/training, parallelism strategies, fault tolerance) describes the infrastructure layer that production meta-harnesses must orchestrate. The GRPO variants taxonomy is particularly relevant for researchers designing reward-shaping harnesses that select or compose RL algorithms. The vLLM/PagedAttention and speculative decoding coverage documents the inference engine primitives that agentic harnesses depend on for cost-efficient multi-turn execution.