Skip to content

LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LICORICE is a label-efficient training scheme for concept-based interpretable RL that reduces the concept annotation budget from millions of labels to ≤500–5000 by combining iterative training, data decorrelation, and disagreement-based active learning. The paper also systematically evaluates VLMs (GPT-4o) as automated concept annotators, finding them effective in simple environments but unreliable for complex ones. The core claim is that interpretable RL policies can match unconstrained baselines using 2000–13000× fewer concept labels.

Problem

Prior concept bottleneck RL methods (e.g., CPM) assume continuous access to ground-truth concept annotations during training, requiring 1M–15M labels per environment. Human annotators cannot label millions of real-time RL states; automated VLM-based labeling incurs prohibitive API costs and inference time. No prior work addresses the concept annotation budget problem in interpretable RL.

Method

LICORICE inserts a concept bottleneck layer into a PPO policy (Ï€(s) = f(g(s))) and trains it with three components:

  1. Iterative training: Alternates between (a) training the concept predictor g on labeled data from the current policy, and (b) freezing g and training the action predictor f via standard PPO. This keeps concept learning on-distribution as the policy improves.

  2. Data decorrelation: Collects an unlabeled candidate set Uₘ of size τ·Bₘ using a per-state random acceptance probability p to break temporal correlations in trajectory rollouts, leveraging the Markov mixing property.

  3. Disagreement-based active learning: Trains an ensemble of N concept models on currently-labeled data; selects the bₘ most informative unlabeled states by variation ratio (classification) or variance (regression) across ensemble members.

For complex environments, a final iteration trains a new action network f' from scratch with a KL-divergence anchoring penalty to the previous policy to avoid local optima. VLM annotation is evaluated by replacing the human oracle with GPT-4o queries using task-specific prompts.

Key Contributions

  • First formulation of the concept annotation budget problem in interpretable RL.
  • LICORICE algorithm combining iterative training, data decorrelation, and ensemble-based active learning.
  • Empirical evaluation across five image-based environments (PixelCartPole, DoorKey, DynamicObstacles, Boxing, Pong).
  • Systematic study of VLMs (GPT-4o) as concept annotators, characterizing when they succeed and fail.
  • Demonstration of test-time concept intervention support.

Results

  • vs. budget-unconstrained CPM: LICORICE achieves ~100% of maximum reward in all five environments using ≤500 labels (simple envs) and ≤5000 labels (complex envs) vs. CPM's 1M–15M labels.
  • vs. budget-constrained baselines (Sequential-Q, Disagreement-Q, Random-Q) at same budget:
  • PixelCartPole: LICORICE 100% vs. best baseline 36% of max reward.
  • DoorKey: LICORICE 99% vs. best baseline 89%.
  • DynamicObstacles: comparable to baselines.
  • Boxing, Pong: outperforms across most budgets.
  • VLM annotation: GPT-4o achieves reasonable concept error in DoorKey (0.30) and DynamicObstacles (0.13), supporting 83%/88% of max reward; fails in PixelCartPole (continuous concepts), Boxing (160–210 discrete values per concept), and Pong (multi-frame physics concepts).
  • Ablation: All three components contribute; removing iterative training (LICORICE-IT) causes the largest drop in PixelCartPole (100%→53%), removing active learning (LICORICE-AC) causes largest drop in DynamicObstacles (99%→58%).
  • Test-time concept intervention monotonically increases reward as more concepts are corrected toward ground truth.

Limitations

  • VLM annotation fails for continuous, high-cardinality discrete, or physics-based concepts; VLM capabilities must improve before replacing human annotators in safety-critical settings.
  • Budget allocation is uniform across iterations (Bₘ = B/M); adaptive allocation is not explored.
  • Evaluated only on five relatively small/controlled environments; scalability to large open-world RL settings is unverified.
  • Adding noise via VLM labels can cause non-monotonic reward improvement as budget increases.
  • Anchoring policy trick for complex environments adds implementation complexity and is heuristic.

Relevance to Vision-Language Models

This paper is directly relevant to VLM-as-a-tool research: it provides a rigorous empirical characterization of when GPT-4o can serve as a zero-shot concept annotator from visual observations, revealing systematic failure modes tied to concept cardinality, continuity, and temporal reasoning. For researchers using VLMs as structured labelers or grounding modules in embodied/agentic systems, the finding that VLMs succeed on discrete, low-cardinality, spatially-grounded concepts but fail on physics quantities and high-resolution coordinates is actionable. The work also connects to the broader line of VLM-grounded interpretability: LICORICE's framework could in principle use stronger VLMs or chain-of-thought prompting to expand the set of concepts VLMs can reliably label. As VLM capabilities improve, this annotation bottleneck may dissolve, making LICORICE a bridge architecture between human-labeled and fully-automated interpretable RL.