Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ACL-LFT introduces a MARL framework where a central agent dynamically selects the optimal context length for decentralized agents at each timestep, using Fourier-based low-frequency truncation to filter redundant historical information. This eliminates the fixed-context-length assumption that plagues prior MARL methods. The framework achieves state-of-the-art performance on PettingZoo, MiniGrid, GRF, and SMACv2 benchmarks.

Problem¶

Deep MARL methods condition policies on large fixed context windows to handle non-Markovian, long-term-dependency environments. Fixed context lengths are either too short (losing relevant history) or too long (injecting noise, increasing computation, hindering exploration). No prior MARL framework adaptively selects context length at runtime; static approaches from NLP (LLM-based prompt tuning) do not transfer to MARL's decentralized, reward-driven setting.

Method¶

ACL-LFT has three coupled components:

Fourier-based Low-Frequency Truncation (LFT): Historical state sequences are transformed via Discrete Fourier Transform; a Dyadic Partition of Unity (grounded in Littlewood–Paley theory) selects only low-frequency bands (indices k ≤ 2^m or k ≥ N−2^m), discarding high-frequency noise. The truncation level m is a tunable hyperparameter and is itself the action space of the central agent.
Central Agent: A separate RL agent that receives the LFT-compressed history as its state and outputs a truncation level m_i ∈ {m_1,...,m_M} as its action. Its reward is a multi-head attention-weighted aggregation of decentralized agents' rewards — attention weights computed from decentralized agents' value/policy representations as keys and the central agent's representation as query — ensuring the central agent's incentive aligns with collective performance.
Spatio-Temporal Decoupling: The central agent is trained independently (optimizing temporal context selection), while decentralized agents are jointly trained on the filtered context plus their current local observations. This avoids the combined high-dimensional search space of joint optimization.

A theoretical regret bound (Theorem 1) shows the cumulative reward advantage of adaptive over fixed context length is Ω(T) − O(T^α) with 0 ≤ α < 1, growing unboundedly with T.

Key Contributions¶

First MARL framework to simultaneously address both computational cost and input representation challenges of long context via adaptive, runtime context length selection.
Fourier + Dyadic Partition of Unity truncation scheme for extracting global temporal trends from decentralized agent histories, with approximation error O(1/t).
Central agent with attention-based reward formulation that aligns context selection incentives with decentralized agent performance.
Formal lower bound proving long-term regret advantage of adaptive over any fixed context length in dynamic environments.
Empirical validation under purely decentralized (no cross-agent history sharing) settings, confirming performance gains are not artifacts of centralized information aggregation.

Results¶

vs. sequence-processing baselines (Transformer, ToST, AMAGO): ACL-LFT outperforms all four environments (Sample Spread, MiniGrid Soccer, GRF 3v1, GRF Counterattack-Hard); performance gap widens with scenario complexity.
vs. fixed context lengths (8, 16, 32, 64 steps) on GRF: ACL-LFT surpasses all; notably, longer fixed lengths (32, 64) underperform shorter ones (8, 16), confirming that noise from excess history hurts.
MiniGrid case study: ACL-LFT scores first goal at step 41 vs. Transformer (step 47), ToST (step 56), AMAGO (step 59); dynamically selects short contexts (length 1–4) near goal-scoring phases.
Without cross-agent history (SMACv2): ACL-LFT achieves 78.9±2.8 / 52.7±4.2 / 77.9±5.3 on 3s5z_vs_3s6z / 5m_vs_6m / corridor vs. AMAGO 76.1±2.9 / 48.1±4.0 / 74.3±4.8 and Mamba 72.6±3.2 / 46.2±4.5 / 69.0±5.9.
Ablation: Removing ACL causes the largest performance drop; removing LFT (ACL-LFT-NO-LFT) shows intermediate degradation; both components are synergistic.

Limitations¶

The central agent adds a second RL training loop; convergence and hyperparameter sensitivity of this meta-level agent are not deeply analyzed.
The action space M (number of discrete truncation levels) requires manual specification; no principled method for setting M or the truncation frequency bound is given.
All experiments use MAPPO as the backbone for decentralized agents; generalization to value-decomposition methods (QMIX, QPLEX) is relegated to an appendix without full ablation.
The Fourier representation discards all high-frequency information globally; this may be harmful in tasks where fast transients are decision-relevant.
Theorem 1's α parameter is described as "non-deterministic," with formal definition deferred to appendix — the bound's practical tightness is unclear.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses a core challenge in multi-agent systems: how much historical context each agent should condition on, and how to represent that context efficiently — problems that recur in LLM-based agent pipelines (e.g., context window management, memory compression). The idea of a dedicated meta-agent that dynamically controls the information horizon for downstream agents is architecturally analogous to memory-controller or context-management modules being explored in LLM agent frameworks. The Fourier-based compression provides a structured alternative to neural summarization (e.g., recurrent memory tokens) for filtering temporal redundancy in long-horizon tasks. For researchers building multi-agent LLM systems that operate over long episode horizons, ACL-LFT offers both a theoretical grounding (Littlewood–Paley, regret bounds) and an empirical blueprint for adaptive context management.