Skip to content

An Information Theoretic Perspective on Agentic System Design

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Christopher Ré (Stanford) + Linderman; first info-theoretic framework for multi-LM agentic systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes an information-theoretic framework for designing compressor-predictor agentic systems, introducing a mutual information (MI) estimator to quantify compression quality in a task-agnostic way. The key finding is that scaling the compressor LM is substantially more effective than scaling the predictor, and that MI rate strongly predicts downstream performance (\(r = -0.84\), \(R^2 = 0.71\)). A 3B local compressor can recover 99% of frontier-LM accuracy on Deep Research benchmarks at 26% of API costs.

Problem

Designing multi-LM agentic systems (e.g., summarizer → reasoner pipelines) is currently trial-and-error: practitioners cannot attribute performance to the compressor's distillation quality vs. the predictor's reasoning without expensive, task-specific end-to-end sweeps. There is no task-agnostic metric to evaluate the compressor in isolation.

Method

The paper models the compressor LM as a noisy channel in a two-stage pipeline \(X \xrightarrow{p(z|x)} Z \xrightarrow{p(y|z)} Y\). It introduces an unbiased Monte Carlo estimator of conditional mutual information \(I(X; Z \mid Q)\):

\[\hat{I}(X; Z) \approx \frac{1}{NM} \sum_{i=1}^{N} \sum_{j=1}^{M} \left[ \log p(z_{ij}|x_i) - \log \frac{1}{N} \sum_{l=1}^{N} p(z_{ij}|x_l) \right]\]

This requires only per-sequence log probabilities from a proxy LM (7–8B scale, different family), making it compatible with accelerated inference engines like SGLang without full vocabulary access. Rate-distortion analysis then fits decaying exponential curves relating information rate \(R = I(X;Z|Q)/L\) (bits per output token) to distortion \(D = 1 - \text{ACC}(Z)\). A meta-analysis via logistic regression ranks the importance of: compressor family > compressor size > predictor size.

Key Contributions

  • Task-agnostic MI estimator for evaluating compressor quality without end-to-end evaluation
  • Empirical demonstration that scaling the compressor dominates scaling the predictor across 5 datasets and 3 model families
  • Discovery that larger compressors are simultaneously more accurate and more concise, causing FLOPs-per-generation to scale sublinearly with compressor size
  • Rate-distortion characterization showing information rate correlates strongly with downstream accuracy and perplexity (\(r = -0.84\), \(R^2 = 0.71\))
  • Design principle: "front-load" compute into on-device compressors to reduce reliance on large cloud-hosted predictors
  • Applied validation on DeepResearchBench: 3B local compressor recovers 99% frontier accuracy at 74% cost reduction

Results

  • Compressor vs. predictor scaling (LongHealth): scaling Qwen-2.5 compressor 1.5B→7B improves accuracy by 60%; scaling predictor 70B→405B yields only 12% improvement
  • Conciseness: 7–12B compressors are up to 4.6× more token-efficient than 1–1.5B within the same family
  • Compute efficiency (Qwen-2.5): scaling compressor 1.5B→7B increases FLOPs-per-generation by only 1.3%
  • Bit efficiency: 7B Qwen-2.5 conveys 5.4× more MI per token than 1.5B sibling; is 1.6× more accurate and 4.6× more concise
  • MI–perplexity correlation (FineWeb): \(r = -0.84\), \(R^2 = 0.71\)
  • Deep Research (DeepResearchBench): local 3B compressor achieves 99% of frontier-LM RACE score at 26% API cost (74% reduction)
  • Meta-analysis ranking: compressor model family > compressor size > predictor size

Limitations

  • MI estimator uses a proxy model (7–8B, different family) to score log-probabilities; proxy choice can introduce variance and bias (ablated in Appendix E.1.4)
  • Estimator is upper-bounded by \(\log(N)\) and can yield small negative values from finite-sample variance (clipped to zero)
  • Experiments focus on GPT-style non-reasoning architectures; reasoning and MoE LMs (Qwen-3) receive only preliminary treatment
  • Cost comparisons are pegged to August 2025 API prices and are time-sensitive
  • The Deep Research pipeline is simplified (single predictor aggregating multiple compressors); full agentic pipelines with tool use are not evaluated

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core engineering question in multi-agent system design: how to allocate compute across pipeline stages. It provides the first task-agnostic proxy metric (MI rate) that can guide compressor selection without end-to-end sweeps, making it highly practical for practitioners building RAG, Deep Research, or any retrieve-then-reason agentic workflow. The "front-load compute into local compressors" principle has immediate architectural implications for edge-cloud hybrid agents, challenging the default assumption that the reasoning (predictor) model is the bottleneck. The information bottleneck framing also connects agent communication channels to a well-developed theoretical literature, opening paths toward principled compression objective design.