ELITE: Experiential Learning and Intent-Aware Transfer for Self-improving Embodied Agents¶

🕒 Published (v1): 2026-03-25 07:25 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ELITE is an embodied agent framework that builds a persistent, evolving strategy pool from execution trajectories via self-reflective distillation, then retrieves relevant strategies at inference time using chain-of-thought embedding similarity. Without any gradient updates, it achieves +9% and +5% success rate over base VLMs on EB-ALFRED and EB-Habitat respectively. In a supervised setting it outperforms RL-trained 3B–7B models using only a 72B VLM with no weight changes.

Problem¶

VLMs trained on static vision-language corpora lack procedural knowledge of physical environments: they skip precondition steps, propose physically invalid actions, and repeat the same errors across episodes because no mechanism exists to extract reusable knowledge from their own interaction history and apply it to structurally similar future tasks.

Method¶

ELITE maintains an evolving strategy pool \(\mathcal{P} = \{(e_i, z_i)\}_{i=1}^M\) initialized empty. Two components operate in a closed loop:

Self-Reflective Knowledge Construction (post-episode): - A Reflective Experience Distiller analyzes completed trajectory \(\tau = \langle(o_0,a_0),\dots,(o_T,a_T)\rangle\) with outcome-conditioned prompts, extracting success patterns + repeatable steps (success) or failure summaries + avoidance guidelines (failure). - A Context Consolidator applies structured delta operations (ADD / REVISE / REMOVE) to integrate reflections into the pool while preserving coherence and removing redundancy.

Intent-Aware Retrieval (pre-episode): - A coarse planner generates a high-level plan \(p = \textsc{CoarsePlanner}(L, o_0)\) that identifies key objects and action sequences. - \(p\) is encoded with BGE-M3 into a 1024-d query vector \(q = \textsc{Embed}(p)\). - Top-\(k\) entries are retrieved by cosine similarity \(M_k = \arg\max_{S \subset \mathcal{P}, |S|=k} \sum_{(e_i,z_i)\in S} \frac{q \cdot z_i}{\|q\|\|z_i\|}\), where pool embeddings \(z_i\) were computed from the planning trace (not the task description) at insertion time. - Retrieved strategies are injected as a structured knowledge section into the task planner's prompt.

In the supervised setting, the distiller additionally performs comparative reflection against ground-truth demonstrations.

Key Contributions¶

An embodied agent framework that continuously accumulates procedural knowledge from binary task outcomes alone, with no parameter updates.
Intent-aware retrieval via CoT-plan embeddings (BGE-M3), enabling transfer between tasks with different surface descriptions but shared execution structure.
A structured strategy pool with ADD/REVISE/REMOVE consolidation to maintain quality as pool size grows.
State-of-the-art results on EB-ALFRED and EB-Habitat in both unsupervised online and supervised generalization settings, outperforming gradient-based baselines without fine-tuning.

Results¶

Online (unsupervised): ELITE (Qwen2.5-VL-72B) achieves 61% avg success on EB-ALFRED (+9 over base VLM at 52%); ELITE (InternVL3-78B) achieves 67% avg on EB-Habitat (+5 over base at 62%).
ELITE outperforms ESCA (scene-graph augmentation) by 23 points on EB-ALFRED (61% vs 38%); ESCA actually underperforms the base VLM (38% vs 52%).
Supervised setting (EB-ALFRED): ELITE achieves 70.8% avg (+21 over base Qwen2.5-VL-72B at 49.8%), surpassing Claude-3.5-Sonnet (66.4%) and ERA (65.2%, trained with RL).
Generalization to unseen task categories: Common Sense 68%, Spatial Awareness 62%, outperforming all open-source and proprietary baselines.
Ablation: removing intent-aware retrieval drops to 56% avg on EB-ALFRED; removing consolidation drops to 55%; CoT retrieval (72.96% task progress on long-horizon) beats TF-IDF on strategy content (68.88%), TF-IDF on instruction (66.84%), all-strategies (68.50%), and random (62.00%).
Long-horizon task progress and success rate improve monotonically as pool size grows across episodes.

Limitations¶

Pool grows unboundedly; no analysis of retrieval latency or memory cost at large scale.
Consolidator is itself a large VLM call (72B) after every episode, making the post-episode overhead substantial in resource-constrained settings.
Cold-start phase (empty pool) yields suboptimal performance with no mechanism to accelerate early learning.
Evaluation limited to two household simulation benchmarks (AI2-THOR, Habitat 2.0); transfer to real-world or non-household domains is unverified.
Binary reward signal only; dense feedback could potentially yield better distillation quality but is not explored.
No analysis of pool staleness or strategy conflict when the environment distribution shifts over time.

Relevance to Harnesses / Meta-Harnesses¶

ELITE is a textbook instance of a meta-harness for agentic loops: it wraps a base VLM agent with a persistent, structured knowledge layer that is updated by a post-episode reflection sub-agent and queried by a pre-episode retrieval sub-agent, forming a closed control loop around the inner policy. The ADD/REVISE/REMOVE consolidation logic is structurally analogous to the kind of evolving configuration/memory store that harness frameworks maintain across runs. The intent-aware retrieval—embedding a generated plan rather than the raw task description—is a generalizable design principle for context injection in any harness that selects tools, prompts, or sub-agents based on the current task's procedural intent. For researchers building meta-harnesses, ELITE demonstrates concretely how to operationalize "learn from prior runs" as a first-class component: structured distillation post-run, vector-indexed storage, and similarity-based retrieval at the start of the next run.