Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Hierarchical preference learning addresses granularity mismatch in long-horizon agent alignment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HPL addresses the granularity mismatch in preference-based alignment of LLM agents: trajectory-level DPO is too coarse for credit assignment, while step-level DPO is statistically noisy. HPL introduces a three-level DPO framework (trajectory, action group, step) with a dual-layer curriculum scheduler that progressively exposes the agent to sub-tasks ordered by group length and reward-gap difficulty.

Problem¶

Applying DPO to long-horizon agent tasks creates a granularity mismatch: trajectory-level DPO (e.g., ETO) yields stable but credit-assignment-blind signals, while step-level DPO (e.g., IPR) provides fine-grained supervision but suffers from high variance under limited Monte Carlo rollout budgets—per-step estimates cannot capture the joint value of multi-action sub-tasks. Neither extreme handles intermediate-length, semantically coherent sub-tasks effectively.

Method¶

HPL operates in two stages after an initial behavior cloning phase (on expert trajectories $\tau^*$):

Stage 1 — Hierarchical Contrastive Data Generation. A frozen reference policy $\pi_\text{ref}$ runs one-shot exploration to collect trajectories. Three preference datasets are constructed: - Trajectory-level $\mathcal{D}_\text{traj}$: expert vs. full failed trajectory. - Step-level $\mathcal{D}_\text{step}$: expert suffix $\tau^w_{t:n}$ vs. agent suffix $\tau^l_{t:m}$ conditioned on shared prefix $\tau_{<t}$ (following IPR). - Group-level $\mathcal{D}_\text{group}$: expert trajectory segmented via $f(\cdot)$ into action groups $\{G^{w,i}\}$; for each, a length-constrained losing group $G^{l,i}$ is sampled from $\pi_\text{ref}$. Group rewards are estimated by MC rollouts: $\hat{r}(G_i) = \frac{1}{M}\sum_{j=1}^M R(\tau_i^{(j)})$.

Four segmentation strategies are compared: Fixed-N (equal-count split), Fixed-K (fixed window size), Uncertainty-based (entropy threshold $H(\pi_\text{ref}(\cdot|o_t)) > \epsilon$ marks boundaries), and Semantic (GPT-4o partitions by sub-goal).

Stage 2 — Multi-Granularity Preference Optimization with Dual-Layer Curriculum. A 3×3 difficulty matrix bins group-level pairs by group length $L$ (sub-task complexity) and $\Delta R = \hat{r}(G^w) - \hat{r}(G^l)$ (sample discriminability). Training proceeds in three phases expanding from bucket $B_{1,1}$ → $B_{1,1} \cup B_{1,2} \cup B_{2,1}$ → all $\bigcup_{L,D} B_{L,D}$. The final loss is: $$\mathcal{L}_\text{final} = \mathcal{L}_\text{BC} + \mathcal{L}_\text{traj-DPO} + \mathcal{L}_\text{step-DPO} + \mathcal{L}^{(s)}_\text{group-DPO}$$ where $\mathcal{L}^{(s)}_\text{group-DPO}$ is computed over the curriculum-stage-appropriate subset $\mathcal{D}^{(s)}_\text{group}$.

Key Contributions¶

Formulation of the granularity mismatch problem in DPO-based agent alignment and a three-level (trajectory/group/step) DPO framework to resolve it.
Group-level DPO with a theoretical bias-variance proposition: with group length $k = \Theta(\log(1/\epsilon))$, variance improves by $\Omega(T/\log(1/\epsilon))$ over step-level DPO while incurring at most additive bias $\epsilon$.
Dual-layer curriculum scheduler over a 2D (group length × $\Delta R$) difficulty matrix—first work to apply structured curriculum to action-group preference optimization.
Systematic comparison of four action grouping strategies including a semantic LLM-based segmenter.

Results¶

Evaluated on ALFWorld (seen/unseen), WebShop (avg. reward / success rate), and InterCode-SQL with Qwen2.5-1.5B and Qwen2.5-7B base models against SFT, RFT, ETO, and IPR.

HPL (Semantic), Qwen2.5-1.5B: average score 59.44 vs. IPR 55.43 and ETO 53.79 (best prior DPO method); ALFWorld unseen 74.13 vs. IPR 66.42.
HPL (Fixed-K(3)), Qwen2.5-7B: ALFWorld seen 85.71, a notable jump over IPR's 73.10; average 65.XX (table truncated at "65.").
HPL (Semantic), Qwen2.5-7B: ALFWorld unseen 84.08 vs. IPR 78.11; WebShop avg. reward 62.97 vs. IPR 62.01; average best among semantic variants.
HPL consistently ranks 1st or 2nd across all benchmarks and both model sizes; the best HPL variant varies by task (Fixed-K on ALFWorld seen/7B, Semantic on most others).

Limitations¶

Group-level reward estimation requires additional MC rollouts from $\pi_\text{ref}$, increasing data generation cost proportional to the number of groups per trajectory.
Semantic segmentation depends on an external powerful LLM (GPT-4o), introducing API cost and a potential quality bottleneck.
The one-shot exploration assumption (no further environment interaction) limits adaptation when $\pi_\text{ref}$ produces low-diversity or low-quality trajectories.
The curriculum hyperparameters (bucket grid resolution, phase boundaries, entropy threshold $\epsilon$) require tuning and are not systematically derived.
Experiments use only Qwen2.5-1.5B and 7B; generalizability to larger or different model families is untested.

Relevance to Agentic AI / LLM Agents¶

HPL directly advances the core problem of credit assignment in post-training LLM agents for long-horizon tasks, a bottleneck in deploying capable autonomous systems. Its group-level DPO formalizes the intuition that agents must reason at the sub-task granularity—aligning with the broader shift from single-turn RLHF to multi-step process supervision. The dual-layer curriculum addresses sample efficiency in offline preference learning, which is practically important when environment interaction is expensive. This work bridges trajectory-level alignment methods (ETO) and process reward models (IPR) into a principled hierarchical framework relevant to web agents, embodied agents, and tool-use agents.