Skip to content

Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performance

🕒 Published (v1): 2025-11-17 00:21 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces NEURO-LOOP, a framework for Reinforcement Learning from Neural Feedback (RLNF) that replaces explicit human feedback in RLHF with passive fNIRS brain signals. The authors collect a 25-participant dataset across three RL domains and train classifiers/regressors to predict agent performance quality from prefrontal cortex hemodynamic responses. Results show feasibility (binary F1 up to 0.79 within-subject, 0.67 multi-subject) but reveal cross-subject generalization as the key open challenge.

Problem

Standard RLHF requires active, explicit feedback (preference labels, demonstrations), imposing cognitive burden and limiting accessibility. Implicit alternatives like gaze/gesture tracking still demand conscious physical effort. There is no established pathway for using passive, zero-effort neural signals (brain activity) as the reward/evaluation signal in RL agent training.

Method

NEURO-LOOP records fNIRS signals from the prefrontal cortex while participants either passively observe or actively control agents in three OpenAI Gym environments (Lunar Lander, Flappy Bird, Robot Fetch and Place). Agent behavior is labeled at three granularities: binary (optimal vs. suboptimal), multi-class (optimal/suboptimal/worst-case), and continuous (deviation from near-optimal policy). Continuous labels use a multi-policy agreement system comparing the agent's chosen action against K=10 near-optimal policies via KL divergence (discrete) or Euclidean distance (continuous). Raw fNIRS phase and intensity signals are band-pass filtered (0.001–0.2 Hz), converted to oxy-/deoxy-hemoglobin, and windowed (5–7 s windows, 1–2 s stride) with statistical features (slope, mean, std, intercept, skewness, kurtosis) extracted per channel. Four classifiers (SVM, KNN, Random Forest, MLP) and three regressors are trained under single-subject, multi-subject, and fine-tuned paradigms; fine-tuning uses ~20% of target-participant data to adapt multi-subject models.

Key Contributions

  • Novel public dataset: synchronized fNIRS recordings + agent transition variables from 25 participants across 3 RL domains (passive and active conditions).
  • Demonstration that fNIRS signals can distinguish 2–3 levels of agent performance quality beyond binary, enabling finer-grained implicit feedback.
  • Multi-policy agreement system for computing continuous, KL-divergence/Euclidean-distance-based performance error labels relative to an ensemble of K=10 near-optimal policies.
  • Cross-subject generalization analysis: zero-shot transfer fails broadly; fine-tuning with 20% subject-specific data recovers +17% binary F1 and +41% multi-class F1 on average.
  • Passive vs. active condition benchmark establishing that passive observation is sufficient for neural signal utility despite lower self-reported cognitive engagement.

Results

  • Single-subject models: binary F1 = 0.79, multi-class F1 = 0.75 (within-subject, best condition).
  • Multi-subject MLP (binary): F1 range 0.61–0.72 across conditions; best on Robot Passive (0.72 ± 0.005).
  • Multi-subject MLP (multi-class): F1 range 0.40–0.54; exceeds random chance (0.33) in all but Flappy Passive.
  • Cross-subject (zero-shot): mostly near random chance; best Robot Passive binary F1 = 0.54 ± 0.01.
  • Fine-tuned models: average +16.9% binary F1, +41.3% multi-class F1 over cross-subject baseline using only 20% target-participant data.
  • Regression (multi-subject): R² = 0.77 (passive), R² = 0.81 (active); cross-subject R² near 0.01; fine-tuning improves Lunar Active to R² = 0.23.
  • NASA-TLX: active tasks elicit higher self-reported workload across all six dimensions; passive tasks described as "boring," potentially degrading signal quality.

Limitations

  • Cross-subject generalization is poor; zero-shot transfer fails in nearly all conditions, requiring per-user calibration data.
  • Dataset imbalance (shorter episodes → fewer samples in some classes) required downsampling, reducing available training data.
  • Low engagement in passive conditions reduces neural signal quality; participants found passive Lunar Lander and Flappy Bird observation unengaging.
  • fNIRS is susceptible to motion artifacts, signal noise, and high inter-subject variability inherent to hemodynamic response.
  • Regression models do not generalize cross-subject without substantial target data; fine-tuning only fully succeeds in one condition (Lunar Active).
  • Study limited to prefrontal cortex; only 25 participants; no real-time closed-loop RLHF integration demonstrated.

Relevance to Agentic AI / LLM Agents

This work directly addresses the feedback bottleneck in RLHF—the dominant paradigm for aligning LLM-based agents—by proposing a passive, zero-effort neural channel as a substitute for explicit human annotation or preference labeling. For agentic systems where sustained human oversight is required but costly, RLNF offers a pathway to continuous, low-burden alignment signals that capture internal human assessments not expressed in overt behavior. The multi-policy agreement labeling scheme is also directly applicable to evaluating agentic action quality in multi-step task settings where ground-truth rewards are unavailable. The cross-subject generalization problem identified here maps directly onto the personalization challenge in deploying aligned agents at scale across diverse users.