🆕 Today's Digest — Agentic AI / LLM Agents¶
New papers/posts collected 2026-07-24.
16 new today
đź“‹ In brief¶
This batch is dominated by two intertwined concerns: the inadequacy of existing evaluation infrastructure and the fragility of agents under realistic conditions. On the evaluation side, at least half the papers introduce new benchmarks specifically designed to expose gaps that prior setups miss — fuzzy requirements, hidden constraints, branching dependency chains, adversarial retrieval environments, and domain-expert financial documents — reflecting a field-wide reckoning that current metrics systematically overestimate agent capability. A second strong thread runs through training: multiple papers tackle how to instill competence via RL, revealing a recurring pitfall where reward design backfires (dense shaping rewards collapsing into dark-room absorbing states, learned text policies failing to outperform fixed prompts) and proposing correctives such as auxiliary loss channels, policy-aware disposable scaffolds, and recursive self-improvement loops. Cutting across both threads is a growing emphasis on memory, context management, and reliability — papers on MemTools, AttriMem, ArbiGraph, and the misleading-knowledge vulnerability all converge on the same finding: agents that look competent on isolated tasks break down when they must maintain coherent state, propagate evidence across long horizons, or resist a single misleading document.
đź“ť Today's papers¶
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders¶
ICAE-Bench is a 480-task benchmark for evaluating coding agents under interactive, "vibe-coding" settings where agents must resolve fuzzy product requirements through multi-turn clarification with an automated User Agent before building a complete repository from scratch. Unlike prior benchmarks that supply fully specified instructions, ICAE-Bench derives controlled ambiguity from real open-source repositories and evaluates generated repos via language-agnostic black-box tests. Experiments across six coding models and two agent frameworks reveal that agents can reproduce visible behavior but consistently fail on hidden constraints, boundary cases, and long-horizon integration.
AREX: Towards a Recursively Self-Improving Agent for Deep Research¶
AREX is a family of Recursively Self-Improving (RSI) deep research agents that exploit the discovery–verification asymmetry in multi-constraint research tasks: rather than searching longer, AREX alternates between an inner evidence-gathering loop and an outer constraint-wise audit loop that converts partially verified answers into better-targeted follow-up problems. Trained on verified synthetic tasks via multi-stage agentic mid-training and long-horizon RL with key-step emphasis, AREX-Base (122B-A10B MoE) and AREX-Turbo (4B dense) outperform comparable-scale baselines across deep-search, wide-search, and reasoning benchmarks.
Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text¶
Existing spatial reasoning benchmarks assume text or coordinate outputs, creating an answer-interface mismatch that prevents image-generation models from being evaluated under the same metrics as text-output VLMs. ProVisE (Protocolized Visual Evaluation) resolves this by constraining image-generation outputs to protocol-defined visual formats and parsing them deterministically into the original benchmark's answer space. A companion benchmark, SpatialGen-Bench, provides 470 samples across 14 subtasks at four capability levels to diagnose where each model family excels or fails.
Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers¶
WorldWeaver (W²) is a streaming multi-agent video diffusion model that augments autoregressive rollout with explicit world state registers (WSR) — learnable tokens shared across agents that are persistently updated each generation step. By grounding these registers with agent status, bird's-eye-view maps, and scene text supervision, the model achieves significantly better logical consistency and cross-agent coherence than approaches that rely solely on local frame history.
OpenForgeRL: Train Harness-native Agents in Any Environment¶
OpenForgeRL is an open-source training framework that enables end-to-end RL training of LLM/VLM agents inside real production inference harnesses (Claude Code, Codex, OpenClaw, etc.) across diverse environments. It uses a lightweight proxy to intercept harness model calls and a Kubernetes orchestrator to run rollouts in remote containers, feeding trajectories into standard RL codebases like veRL without train–deploy mismatch.
đź“Ś Why selected: MSR team; trains agents inside real harnesses (Claude Code, Codex); directly actionable for harness builders
PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning¶
PATS reframes external skills as disposable training scaffolds for LLM agent RL rather than persistent artifacts to retain or distill. A dynamic Bank of experience-derived entries expands when a weak policy fails, revises as residual errors shift, and compresses once the policy becomes competent—then is discarded entirely at deployment. On ALFWorld and WebShop, PATS improves over GRPO by up to 18.6 SR points while reducing deployment interaction tokens by ~30–39%.
đź“Ś Why selected: Policy-aware scaffolding for long-horizon RL agents; addresses exploration collapse in sparse-reward settings
GuardianAgentBench: Where Agents Fail and How to Guard Them¶
GuardianAgentBench (GABench) is a 580-scenario benchmark evaluating LLM agent safety on three production frameworks (LangChain, LlamaIndex, Vectara) under five adversarial attack modes. Even the strongest tested configuration (Claude Opus 4.5 on Vectara) achieves only 74.8% overall accuracy. A proof-of-concept execution-time guardrail layer recovers 19.9% of failures at a false positive rate of 0.5%, outperforming system-prompt-based defenses across all models.
📌 Why selected: 580-scenario safety benchmark for autonomous agents; maps failure modes and guards—directly useful for harness evaluation
The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works¶
Dense per-step observation-prediction rewards, when injected into the GRPO reward channel, reliably drive LLM agents into a degenerate "dark room" absorbing state (prediction accuracy →1.0, task success →0) across all tested model scales. The root cause is GRPO's std-normalization of group advantages, which makes bounded shaping rewards produce full-scale gradient pressure in all-fail groups regardless of the shaping coefficient \(\lambda\). Switching the same signal to an auxiliary cross-entropy loss channel recovers ~20 points of task success, and a shuffled-gold placebo demonstrates that exact content–context pairing is not the active ingredient.
📌 Why selected: Dense reward collapse under GRPO for LLM agents; identifies what actually works—critical RL training insight
AttriMem: Attribution-Guided Process Feedback for Agent Memory Learning¶
AttriMem introduces an attribution-guided process-feedback framework for training LLM agent memory-construction policies via RL. Rather than relying on sparse outcome-level rewards, it uses token-level attribution (via ContextCite) to estimate each intermediate memory action's causal contribution to the final answer and converts these scores into dense local process rewards. On long-horizon dialogue QA, it consistently outperforms retrieval-based, heuristic, and RL-based baselines across three benchmarks.
📌 Why selected: Attribution-guided memory learning for agents; addresses what to store/compress/discard—core harness problem
MemTools: A Unified Research Framework for Interoperable Agent Memory¶
MemTools is an interoperability research framework that decouples agent memory pipelines from deployment environments by standardizing the memory lifecycle through declarative data contracts, orthogonally separating evaluation protocols from benchmark datasets, and unifying symbolic, neural, and multimodal memory under a shared computational interface. It enables controlled isolation of individual memory design variables—previously impossible due to pervasive architectural coupling in existing systems. Empirical studies show cross-system hybrid pipelines can outperform native designs.
đź“Ś Why selected: Unified interoperable agent memory framework; targets architectural fragmentation across memory lifecycle stages
Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions¶
Deep Research agents that autonomously plan, retrieve, and synthesize long-horizon research reports are surprisingly vulnerable to factually misleading knowledge embedded in their retrieval environments. Even a single misleading document can cause agents to adopt false conclusions in final reports, and existing verification defenses reduce but do not eliminate this failure mode.
đź“Ś Why selected: Reliability of deep research agents under misleading knowledge; evaluates long-horizon retrieval+synthesis pipelines
DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations¶
(no TL;DR)
NVIDIA-labs OO Agents: Native Python Object-Oriented Agents¶
NVIDIA Object-Oriented Agents (NOOA) is a Python agent framework where an agent is a plain Python class: methods with real bodies are deterministic code, methods with ... bodies become LLM-driven loops, type annotations are runtime-validated contracts, and docstrings are prompts. This "agent-as-object" model eliminates the split across prompt templates, schemas, callbacks, and orchestration DSLs by reusing Python's existing abstractions. Evaluations on SWE-bench Verified, Terminal-Bench 2.0, and ARC-AGI-3 demonstrate that current models effectively use this interface.
ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management¶
ArbiGraph is a benchmark generator that composes parameterized, executable natural-language tasks into verifiable dataflow graphs to evaluate context management in tool-assisted LLM agents. It independently controls context length, dependency topology, and value type, enabling exact automatic verification. Evaluating Qwen3.5-27B reveals that baseline task accuracy does not predict dependent-chain accuracy: math accuracy drops from 94.5% (isolated) to 61.2% on branching multichain topologies.
đź“Ś Why selected: Scalable benchmark for context management in tool-assisted agents; evaluates retain/update/discard across workflows
From Agent Failures to Text Policies: What Works and What Breaks¶
This paper dissects why TextGrad-style text optimization fails when applied to frozen LLM agents. Human-written policies yield a +5.0 success-point gain on TextWorldExpress, proving useful policy text exists and agents can follow it, but policies learned autonomously from trajectories do not reliably outperform fixed prompting even with richer traces, counterfactual branches, or multi-round GEPA search. The bottleneck is generating and selecting good rules from experience, not executing them.
đź“Ś Why selected: Empirical analysis of TextGrad applied to agents; identifies what breaks in text-gradient optimization for agent components
FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents¶
FinanceComplexQA is a bilingual (EN/CN) benchmark of 2,026 expert-level open-ended QA tasks over 1,009 industrial-grade financial documents, requiring dual-context reasoning (explicit document evidence + implicit domain knowledge) and cross-layout evidence fusion. Alongside the benchmark, the authors release Finance-LaTeX SKILL, a multi-agent synthesis pipeline that generates 2,000 professional financial documents and 6,000 QA pairs with layout-aware LaTeX structure. Evaluation of RAG, MCP, and agentic systems shows a large performance gap between current systems and professional financial analyst capability.