Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces 3M-Progress (Model-Memory-Mismatch Progress), a model-based intrinsic motivation algorithm that drives autonomous exploration by measuring KL divergence between an online world model and a frozen ecological prior. Trained without any behavioral or neural supervision, a virtual zebrafish agent using 3M-Progress replicates both the ethological active/passive behavioral cycling and the whole-brain neural-glial calcium dynamics recorded in larval zebrafish during futility-induced passivity.

Problem¶

Existing model-based intrinsic motivation methods (ICM, RND, Disagreement, γ-Progress) suffer from non-stationarity: their reward landscapes flatten as world-model training loss converges, preventing convergence to a stable exploratory policy and causing agents to either perseverate on unpredictable stimuli or collapse to a single behavioral mode. Simultaneously, systems neuroscience lacks goal-driven, population-level computational models of neural-glial computation during naturalistic, task-independent behavior.

Method¶

3M-Progress decouples the two memories used in γ-Progress: a fixed ethological prior ωθ pretrained in a naturalistic environment (zebrafish swimming with fluid dynamics), and an online world model ωθ′ trained on the deployment environment. Both are Gaussian forward-dynamics models in sensory feature space. The model-memory-mismatch signal is the KL divergence ϵt = DKL[ωθ ‖ ωθ′], fed through a leaky integrator ϵ̂t = (1−γ)ϵ̂t−1 + γϵt. Intrinsic reward is rᵢ = |ϵ̂t − ϵt|, which reinforces transitions between niche-seeking (memories agree) and niche-avoiding (memories diverge) partitions of state-action space, yielding a stationary policy objective since ωθ is frozen. The embodied agent is a 6-link MuJoCo swimmer with LSTM actor-critic trained via PPO; the intrinsic drive module generates reward but is not used for planning.

Key Contributions¶

3M-Progress algorithm: stationary intrinsic reward derived from KL divergence between frozen ecological prior and online world model, filtered through a leaky integrator
First autonomous embodied agent (virtual zebrafish) that predicts whole-brain calcium dynamics purely from self-supervised intrinsic optimization, with no behavioral or neural training data
First goal-driven, population-level computational model of neural-glial computation
Normative mapping: the leaky integrator in 3M-Progress is functionally equivalent to radial astrocytes accumulating NE-MO signals; the mismatch detector maps to noradrenergic neurons in lateral medulla oblongata
Virtual zebrafish environment in MuJoCo with fluid dynamics matching experimental parameters from Mu et al. 2019

Results¶

Behavioral alignment: 3M-Progress agents achieve the highest Pearson r between agent swim power (joint torques) and zebrafish swim power (motor neuron activity) at both active→passive and passive→active transitions; all other methods (ICM, RND, Disagreement, γ-Progress, static, homeostatic, MaxEnt) failed to sustain stable behavioral cycling
Neural-glial alignment: 3M-Progress captures nearly all explainable variance in both neural and astrocytic whole-brain calcium responses under a stringent one-to-one unit mapping; inter-animal alignment (pairwise across 11 zebrafish subjects) was ~100%, establishing the ceiling
3M-Progress is the only model simultaneously near-ceiling on both behavioral alignment and whole-brain neural-glial alignment (Figure 4B); data-derived controls (Gaussian process, PID, average population, white noise) and all RL baselines scored substantially lower
PCA of the agent's LSTM core module reveals latent trajectories that mirror the glial accumulation and neural transient patterns observed in zebrafish calcium imaging

Limitations¶

Environments are constrained virtual arenas; ecological realism is limited compared to open water habitats
Biomechanical model abstracts away muscles and motor circuits; low-level controller fidelity is limited
Model operates at population level and does not capture detailed astrocyte biochemistry, morphology, or anatomy
Evaluation restricted to a single autonomous behavior (futility-induced passivity); generalization to richer behavioral repertoires is left to future work
Fixed frozen prior requires pretraining in a well-characterized ecological niche, which may not be available for all target agents

Relevance to Agentic AI / LLM Agents¶

3M-Progress offers a principled, biologically-grounded intrinsic motivation mechanism for reward-free autonomous agents: using a frozen ecological prior to define a stationary goal overcomes the non-stationarity that plagues curiosity-driven exploration, a standing problem in open-ended agentic settings. The framework directly addresses the challenge of building agents that self-direct without external reward or task specification — a core requirement for long-horizon LLM agents operating in open-ended environments. The insight that stable autonomous behavior requires memory of an "ecological niche" against which novel experience is measured maps conceptually onto ideas like in-context behavioral priors or persistent agent memory in LLM architectures. The validation methodology — aligning agent internal representations to biological neural data — also points toward a rigorous evaluation paradigm for assessing whether agentic architectures develop coherent world models.