Skip to content

Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning

🕒 Published (v1): 2026-07-08 18:47 UTC · Source: Arxiv · link

Why this paper was selected

Offline agent alignment via imitation learning regularization — addresses feedback manipulation in agent training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Feedback Manipulation Regularization (FMR) is an algorithm-agnostic regularizer that incorporates evaluative feedback (good/bad labels on state-action pairs) into offline imitation learning to suppress misaligned behaviors. It operates via feedback-induced temperature scaling in probability space, redistributing policy mass toward aligned actions. Applied to Safety Gymnasium environments, FMR achieves up to 98% reduction in misalignment across four baseline imitation learning algorithms without sacrificing task aptitude.

Problem

Standard imitation learning algorithms trained on mixed expert/imperfect demonstration datasets learn misaligned policies even when expert data is fully aligned — and the problem compounds as the expert-to-imperfect data ratio shrinks. Preference-based feedback (RLHF/LfP), the dominant alignment signal in LLM fine-tuning, is fundamentally relative: it cannot declare absolute alignment and is ill-suited to fully sequential decision-making settings where behavioral precision is required. No prior work provides a single-stage, offline, algorithm-agnostic mechanism for fusing evaluative feedback with demonstrations in sequential RL/imitation settings.

Method

FMR extends any imitation learning objective \(\mathcal{L}_{IL}(\theta)\) with a regularization term derived from evaluative feedback \(h: S \times A \to \{-1, 0, 1\}\).

Feedback-adaptive temperature. Each state-action pair receives a per-action temperature via: $\(\tau_j = \begin{cases} \beta^{-h} & \text{if } h < 0 \text{ and } j = \text{selected action} \\ \beta^{h} & \text{if } h > 0 \text{ and } j \neq \text{selected action} \\ 1 & \text{if } h = 0 \end{cases}\)$ with \(\beta > 1\). Negative feedback raises \(\tau > 1\) for the taken action (discourages it); positive feedback raises \(\tau > 1\) for all other actions (sharpens mass onto the selected action).

Feedback-tempered surrogate policy. The aligned policy is approximated as: $\(\pi_\tau(a_j | s) = \frac{\pi_\theta(a_j | s)\, \tau_j^{-1}}{Z_\theta(s)}\)$ This operates in probability space (not logit space), requiring no assumptions on how \(\pi_\theta\) is parameterized.

Regularizer. Aligning \(\pi_\theta\) to \(\pi_\tau\) via reverse KL divergence decomposes to: $\(\mathcal{R}_{\text{FMR}}(\theta) = \mathbb{E}_{s \sim D^U} \mathbb{E}_{a \sim \pi_\theta(\cdot|s)} \left[\log \tau(s, a)\right]\)$ The combined loss is \(\mathcal{L}(\theta) = \mathcal{L}_{IL}(\theta) + \alpha\, \mathcal{R}_{\text{FMR}}(\theta)\).

Credit assignment (TAMER-style 600 ms window) maps feedback to prior states, enabling sub-trajectory targeting. The approach extends naturally to Learning from Noisy Demonstrations (LfND) where imperfect demonstrations receive negative feedback on cost-violating transitions.

Key Contributions

  • Algorithm-agnostic FMR regularizer that integrates evaluative (absolute) feedback into any imitation learning objective via temperature scaling in probability space.
  • Formal derivation showing the regularizer is equivalent to minimizing reverse KL divergence between \(\pi_\theta\) and the feedback-tempered surrogate \(\pi_\tau\), with connection to entropy regularization as a special case.
  • Adapted Safety Gymnasium environments as a principled alignment testbed, using environment cost signals as alignment metrics (not training objectives).
  • Demonstrated robustness in scarce data regimes (expert:imperfect ratios 1:5, 1:2, 1:1) across high- and low-overlap demonstration distributions.
  • Empirical comparison against two alternative feedback-use strategies: DVL (feedback as reward) and CPL (feedback as preference score), showing FMR's superiority.

Results

  • SlowSwim (high overlap, velocity task): up to 98% reduction in misalignment for ReCOIL+FMR vs. ReCOIL baseline, near-zero misalignment achieved; stable across all data ratios.
  • PathM navigation (high overlap): ReCOIL+FMR achieves 92% misalignment reduction; success rate maintained or improved across all ratios.
  • PathBB navigation (low overlap): ReCOIL+FMR achieves 67% misalignment reduction; strongest success rate among all methods at all data ratios.
  • SlowHop: 75% misalignment reduction across algorithms and data ratios.
  • SlowWalk (large action space, |A|=36): up to 76% misalignment reduction; DemoDICE+FMR at 10-50 ratio is an outlier with little gain.
  • FMR+ReCOIL outperforms DVL and CPL across all navigation tasks and data ratios; DVL exhibits poor misalignment despite comparable returns; CPL degrades sharply at the 10-50 ratio.
  • FMR applied to BC (the weakest baseline) still yields consistent misalignment reduction, demonstrating algorithm-agnosticism.

Limitations

  • Requires evaluative feedback annotations on demonstration data; real human feedback collection introduces annotation cost and variability (navigation tasks use a single human evaluator).
  • Velocity task feedback is proxy-generated from cost violations, not real human labels — generalizability of the proxy to true human evaluative feedback is unvalidated.
  • Large discrete action spaces (SlowWalk, \(|A|=36\)) degrade performance for some base algorithms; FMR's gains diminish and require manual tuning (\(\alpha=2\), positive feedback injection on \(D^E\)).
  • DemoDICE+FMR at the 10-50 ratio on SlowWalk shows negligible improvement, suggesting sensitivity to the base algorithm's hyperparameters in low-data regimes.
  • Evaluated only in discrete action spaces; applicability to continuous action spaces is not demonstrated.
  • The framework is offline; online feedback during policy execution (as in TAMER/COACH) is not explored.

Relevance to Agentic AI / LLM Agents

FMR addresses a core challenge in agentic systems: ensuring that sequential decision-making agents adhere to human values when trained offline from mixed-quality demonstrations, directly relevant to deploying LLM-based or RL-based agents in safety-critical tasks. The shift from relative (preference) to absolute (evaluative) feedback reframes alignment as a precision problem rather than a ranking problem — a distinction that becomes critical when agents must avoid specific behaviors rather than merely prefer better ones. The algorithm-agnostic formulation means FMR could augment any offline imitation learning backbone used in tool-use agents, robotic agents, or LLM fine-tuning pipelines extended to sequential settings. Its robustness in scarce aligned-data regimes is particularly relevant for agent alignment, where obtaining high-quality aligned demonstrations is typically the bottleneck.