Skip to content

Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

🕒 Published (v1): 2025-11-19 02:59 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAQ (Macro Action Quantization) is a framework that makes RL agents behave more like humans by distilling human demonstrations into discrete macro actions via a Conditional VQVAE, then constraining RL training to operate within that human-like action space. Applied to IQL, SAC, and RLPD on D4RL Adroit dexterous manipulation tasks, MAQ substantially improves trajectory similarity metrics and fools human evaluators in a Turing Test more convincingly than baseline RL agents.

Problem

Standard reward-maximizing RL agents achieve high task performance but exhibit unnatural behaviors (unnatural grips, spinning, shaking, non-human motion sequences) that make them easily distinguishable from humans. Prior approaches to human-likeness either rely on handcrafted behavioral penalties or pure imitation learning, which either requires laborious rule engineering or sacrifices task performance. No general, plug-in framework existed for constraining RL agents to human-like action manifolds without sacrificing reward optimization.

Method

MAQ frames human-likeness as trajectory optimization within a human manifold, implemented via receding-horizon control (RHC) over macro action segments of length H:

  1. Human behavior distillation: A Conditional VQVAE is trained on (state, macro-action) pairs extracted from 25 human demonstrations per task via sliding window. The encoder maps an H-step action segment conditioned on state to a latent vector; vector quantization snaps it to the nearest entry in a learned codebook of size K; the decoder reconstructs the macro action. Loss: reconstruction + commitment + codebook terms (Eq. 4).

  2. RL with quantized macro actions: The RL policy π_θ operates over the discrete codebook indices (replacing primitive actions). At each state, it selects index k; the VQVAE decoder maps (e_k, s_t) → macro action m̃_t which is executed for H steps in the environment (SMDP formulation). This reduces the exponential macro-action space (n^H) to K codebook entries.

  3. Integration: MAQ wraps any off-the-shelf RL algorithm (IQL, SAC via Discrete SAC, RLPD) unchanged except for the action space substitution. H=9 is the best-performing sequence length.

Key Contributions

  • Formal framing of human-likeness as constrained trajectory optimization with a human manifold, realized via receding-horizon control
  • Conditional VQVAE-based macro action codebook that compresses exponentially large human-like action sequences into K discrete entries
  • Plug-in framework compatible with any RL algorithm, requiring no handcrafted behavioral rules
  • Human evaluation protocol (Turing Test + ranking test) with 19 evaluators on four Adroit dexterous manipulation tasks
  • Demonstration that human-likeness and task performance are not fundamentally at odds

Results

  • Trajectory similarity (DTW/Wasserstein, averaged over 4 Adroit tasks): MAQ raises avg DTWs from 0.21 (IQL)→0.63 (MAQ+IQL), -0.49 (SAC)→0.56 (MAQ+SAC), 0.10 (RLPD)→0.56 (MAQ+RLPD); all normalized scores higher indicate closer alignment with human demonstrations
  • Door task DTWs: IQL 0.43→MAQ+IQL 0.84; SAC -0.39→MAQ+SAC 0.80; RLPD -0.06→MAQ+RLPD 0.76
  • Turing Test win rates (avg over 4 tasks): MAQ+RLPD 39% > MAQ+SAC 34% > MAQ+IQL 32% > BC 24% = RLPD 24% > SAC 19% > IQL 13%
  • Human-likeness ranking test win rates: Human 74% > MAQ+RLPD 71% > MAQ+SAC 61% = MAQ+IQL 61% > BC 43% > RLPD 38% = IQL 38% > SAC 14%
  • Task success rate generally maintained or improved (MAQ+IQL Door: 0.16→0.93; MAQ+RLPD avg: 0.68→0.52, though RLPD baseline was already strong on some tasks)
  • Longer macro action length H consistently improves both human-likeness and task success (H=9 optimal)

Limitations

  • Requires human demonstrations to build the codebook; quality and coverage of demonstrations bound the representable human-like behaviors
  • Codebook size K and macro action length H are hyperparameters requiring tuning; too-small K may fail to cover the behavioral diversity
  • Evaluated only on D4RL Adroit (dexterous manipulation with Shadow Hand); generalization to other domains (e.g., locomotion, navigation, games) not empirically demonstrated
  • The SMDP formulation commits to H-step macro actions before replanning, potentially reducing reactivity in dynamic environments
  • Success rate on some tasks (Hammer with MAQ+IQL/MAQ+SAC: 0.00) shows that the human-like constraint can hurt task performance for harder manipulation tasks

Relevance to Agentic AI / LLM Agents

While focused on motor-control RL rather than LLM agents, MAQ's core idea — constraining an optimizer to operate within a human behavioral manifold extracted from demonstrations — is directly analogous to alignment techniques in LLM agents, where RLHF/Constitutional AI aim to keep LLM behavior within human-acceptable response space. The macro-action abstraction (chunking primitive decisions into higher-level, semantically coherent units) mirrors how LLM agents benefit from tool-use abstractions and sub-task planning rather than raw token-level action selection. The human evaluation methodology (Turing Test framing, 2AFC forced choice) offers a transferable evaluation protocol for assessing whether agentic AI behaves in ways humans find natural and trustworthy — a growing concern in embodied and interactive LLM agents.