Full Bayesian Reinforcement Learning via LF-IBIS¶

🕒 Published (v1): 2026-07-02 05:54 UTC · Source: Arxiv · link

Why this paper was selected

Full Bayesian RL via LF-IBIS; principled uncertainty-aware agent learning with sequential decision-making

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LF-IBIS is a likelihood-free Bayesian reinforcement learning algorithm that combines Approximate Bayesian Computation (ABC) with Iterated Batch Importance Sampling (IBIS) to maintain full posterior distributions over both environment parameters $\mu$ and optimal policies $\pi^*$ without requiring a tractable likelihood. Unlike prior BRL methods that reduce posteriors to point estimates, it propagates uncertainty all the way to the policy level, enabling principled exploration via a Bayesian analogue of Thompson sampling.

Problem¶

Standard Bayesian RL requires an explicit, tractable likelihood $L(\mu; x)$ for the MDP transition dynamics—unavailable when the environment is a black-box simulator or complex computational model. Additionally, existing likelihood-free RL methods (e.g., Dimitrakakis & Tziortziotis) operate offline and only place posteriors over environment models, not over optimal policies, losing information useful for exploration–exploitation decisions.

Method¶

The framework casts RL as statistical inference over MDP parameters $\mu$ governing transition matrices $P(a)$. Under the Maximum Entropy RL objective, the optimal policy has a Boltzmann form: $$\pi^*_\mu(a \mid s) = \frac{\exp\!\left(\tfrac{1}{\lambda}Q^*_\mu(s,a)\right)}{\sum_{a'}\exp\!\left(\tfrac{1}{\lambda}Q^*_\mu(s,a')\right)}$$ making $\pi^*$ a deterministic function of $\mu$, so posterior uncertainty over $\mu$ propagates to $p(\pi^* \mid x)$.

LF-IBIS approximates this posterior online by: 1. Running SMC-ABC (Del Moral et al. 2012) with an adaptive tolerance sequence $\epsilon_1 > \cdots > \epsilon_n$ selected via bisection to maintain target ESS: $\text{ESS}(\omega_i(\epsilon^*)) = \alpha \cdot \text{ESS}(\omega_{i-1}(\epsilon_{i-1}))$. 2. Replacing the exact likelihood with Monte Carlo ABC likelihoods $\hat{f}_{\epsilon,M}(x \mid \mu) = \frac{1}{M}\sum_{m=1}^M \kappa(d(y_m, x); \epsilon)$ using $M$ simulated pseudo-histories per particle. 3. Updating beliefs online via IBIS as new agent–environment interactions arrive, without reprocessing full history.

Policy updating uses a Bayesian Mean Squared Error (BMSE) criterion: given $L$ posterior policy samples $\{\pi^{(\ell)}\}$, the current policy is replaced only if a newly sampled candidate $\tilde\pi$ achieves lower $\text{BMSE}(\pi) = \frac{1}{L}\sum_\ell \|\pi^{(\ell)} - \pi\|^2$.

Key Contributions¶

First online, fully likelihood-free BRL algorithm that places a posterior over optimal policies $p(\pi^* \mid x)$, not just model parameters.
Integration of SMC-ABC (adaptive threshold mechanism) with IBIS for sequential, incremental posterior updates in RL.
Bayesian Thompson-sampling-like exploration driven by posterior policy uncertainty, with exploitation emerging naturally as uncertainty concentrates.
BMSE-based policy update rule for principled exploration–exploitation switching.
Asymptotic analysis of LF-IBIS.
Empirical validation in response-adaptive randomization (clinical trials), where a conjugate closed-form posterior allows direct accuracy comparison.

Results¶

In the response-adaptive randomization experiment (clinical trials with Beta-Bernoulli conjugate model), LF-IBIS posterior approximations closely match the exact conjugate posterior, validating the method quantitatively.
LF-IBIS outperforms standard Rejection ABC and matches the exact posterior in settings with tractable likelihoods, serving as accuracy validation.
Additional experiments demonstrate online policy updating in settings with no closed-form posterior, showing the method correctly updates policies as posterior mass concentrates.
No numerical benchmark tables with specific metric values are reported in the provided text excerpt.

Limitations¶

Scalability: ABC acceptance rates drop as history length $N$ grows, requiring summary statistics $\eta(\cdot)$ that introduce a further approximation beyond the $\epsilon$-tolerance.
Restricted to finite state/action spaces under the homogeneous MDP assumption.
Computational cost scales with number of particles $L$ and pseudo-histories per particle $M$; no wall-clock comparisons to model-free baselines are provided.
BMSE-based policy update rule is heuristic; the choice of when to trigger exploration vs. exploitation remains sensitive to $L$ and the posterior spread.
Validated primarily on low-dimensional bandit/clinical-trial settings; scalability to large state spaces or deep RL domains is not demonstrated.

Relevance to Agentic AI / LLM Agents¶

This work addresses a foundational challenge for any RL-trained agent operating in black-box or simulator-backed environments—how to maintain calibrated uncertainty over both world models and policies without a tractable likelihood—directly relevant to model-based agentic systems that learn from environment interaction. The fully Bayesian treatment of exploration–exploitation via posterior policy distributions provides a theoretically principled alternative to heuristic $\epsilon$-greedy or UCB strategies used in current LLM-based agents. The online, incremental inference design matches the streaming-interaction regime of deployed agents that cannot afford full posterior recomputation after each step. For LLM agent researchers, the ABC framing offers a pathway to Bayesian credit assignment in environments modeled as opaque simulators (e.g., tool-use sandboxes, web browsers), where transition likelihoods are inherently unavailable.