Guava: An Effective and Universal Harness for Embodied Manipulation

🕒 Published (v1): 2026-06-16 18:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Guava is a harness framework for embodied robotic manipulation that systematically identifies three key design principles — iterative ReAct-style perception-reasoning-action loops, semantic action abstractions, and multimodal observations — and uses them to distill frontier VLM capabilities into a 4B open-source model trained on fewer than 2K simulation trajectories, achieving competitive real-world performance with zero-shot sim-to-real transfer.

Problem

Existing harness-based manipulation systems rely on one-shot code generation with powerful frontier models, offering no closed-loop feedback or failure recovery. End-to-end VLA approaches require large robot datasets and are embodiment-specific. It is unclear which harness design choices are necessary for robust long-horizon manipulation, and whether effective harnesses can generalize across model scales including small open-source models.

Method

Guava defines a structured agent-environment interaction protocol built from three validated ingredients:

1. ReAct-style iterative loops: the VLM repeatedly observes, reasons (inside <think> tags), and calls tools, enabling online replanning and failure recovery rather than open-loop trajectory prediction.
2. Semantic action abstractions: a fixed tool API (Table 1) exposes high-level primitives (grasp(object), align(object, direction, clearance), release(), etc.) plus lower-level geometric tools (move(x,y,z), rotate(angle, axis)), delegating motion planning to specialized controllers and freeing the VLM for task decomposition.
3. Multimodal observations: image + textual state descriptions are provided at each step, combining spatial grounding with compact symbolic task-progress summaries.

For distillation, a data generation engine runs frontier VLMs (GPT-5.4) under the Guava harness in Robosuite simulation with scene randomization and targeted execution perturbations to produce both successful and recovery trajectories (~1,934 total, across six task categories). A two-stage training pipeline first applies SFT on all trajectories, then GRPO with a sparse task-success reward on challenging long-horizon tasks. The base model is Qwen3.5-4B with frozen vision encoder and aligner, trained on 8×H100s in bfloat16.

Key Contributions

• Systematic ablation study identifying the three indispensable harness ingredients (workflow, action space, observation modality) via controlled experiments with GPT-5.4 on six Robosuite tasks.
• Guava harness framework: a concrete tool API + ReAct workflow + multimodal observation pipeline for embodied manipulation.
• Data-efficient distillation pipeline using simulation-only trajectories (<2K) with recovery augmentation to train a 4B model (Guava-Agent-4B).
• Demonstration that a well-designed harness is model-agnostic: even a 4B model equipped with Guava surpasses GPT-5.4 overall (75.6% vs. 70.2%) and transfers zero-shot to a real Franka robot.
• Evidence that RL post-training (GRPO) substantially improves long-horizon recovery, boosting shell game from 6.7% (SFT) to 60.0% and place-all-red-objects from 0% to 93.3%.

Results

• Simulation (Table 2, 15 episodes each): Guava-Agent-4B achieves 75.6% overall success vs. GPT-5.4 at 70.2%, CaP-Agent0 at 62.7%, and base Qwen3.5-4B at 23.1%.
• ID tasks: 100% on place-can-in-box and remove-cube-from-tray.
• OOD object: 100% on pick-up-carrot and lemon-in-bin; push-pot 73.3% vs. GPT-5.4's 26.7%.
• OOD long-horizon: 93.3% on separate-food-and-utensils and set-table.
• Hard failures: shell game 6.7%, place-all-red-objects 0% (pre-RL).
• Real world (Figure 4, 10 episodes each): Guava-Agent-4B achieves 86% ID avg and 92% OOD avg, outperforming GPT-5.4 (76% ID, 82% OOD) and CaP-Agent0 (42% ID, 24% OOD).
• RL ablation (Figure 6): RL post-training raises shell game 6.7%→60.0% and place-all-red-objects 0%→93.3%.

Limitations

• Cannot handle dexterous manipulation; the action primitive set restricts fine-fingered control.
• Cannot correct tool-level errors directly (e.g., invalid grasp proposals from SAM3 segmentation failures); recovery is limited to retries and replanning.
• Single fixed-camera viewpoint; occlusions constrain perception.
• Distilled model performance is bounded by the frontier VLM teacher; failure modes of the teacher propagate.

Relevance to Harnesses / Meta-Harnesses

Guava is directly a study of harness design: its central contribution is a controlled ablation over workflow strategy, action-space abstraction level, and observation modality, making it one of the few papers to empirically validate which harness components matter and why. The finding that iterative ReAct loops, semantic tool abstractions, and multimodal context each independently contribute to performance is a concrete design rule for anyone building embodied meta-harnesses. The use of the harness itself as a data engine — generating distillation trajectories for smaller models — generalizes the meta-harness concept: the harness is not just a runtime wrapper but a scaffold for capability transfer. This connects directly to the broader research line of using harnesses to decouple reasoning from execution and to enable model-agnostic, compositional agent architectures.