Skip to content

OrbitZoo: Real Orbital Systems Challenges for Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OrbitZoo is a multi-agent reinforcement learning benchmark for orbital dynamics built on the Orekit high-fidelity astrodynamics library, standardized via PettingZoo for MARL with POMDP structure. It addresses the reality gap in existing RL orbital environments by incorporating realistic perturbative forces (atmospheric drag, SRP, third-body effects, harmonic gravity) and validating against real Starlink ephemeris data (MAPE 0.16%). Experiments spanning Hohmann transfers, collision avoidance, and GEO constellation coordination demonstrate near-optimal RL policy learning under realistic dynamics.

Problem

Existing RL environments for orbital control are built from scratch with simplified physics models (often only Newtonian attraction or J2-only gravity), lack multi-agent support, do not propagate uncertainty, and are rarely validated against real satellite data. This creates a large sim-to-real gap, limits generalization, and hinders reproducible benchmarking across the community.

Method

OrbitZoo wraps Orekit's numerical orbital propagator (Holmes–Featherstone harmonic gravity, atmospheric drag from historical weather data, SRP with Moon occlusion, all-planet third-body forces, Dormand-Prince variable-step integrator) in a Python/PettingZoo-compatible MARL environment. State representations support Cartesian, Keplerian, and equinoctial elements. Thrust actions use a polar parameterization (magnitude T, deviation θ, azimuth ϕ) converted to the RSW frame, modeling time-continuous thrust with fuel constraints. The MARL structure follows MA-POMDP: each agent receives a local observation and acts via a decentralized policy; centralized training with decentralized execution (CTDE) is supported. Bayesian optimization over drag coefficient, reflection coefficient, and satellite radius was used to tune simulation parameters against 31 real Starlink ephemerides from Space-Track. Parallelism from Orekit enables scaling to thousands of bodies.

Key Contributions

  • High-fidelity orbital data generation via Orekit with parallel propagation and support for realistic perturbative forces not available in prior open-source RL frameworks.
  • PettingZoo-standardized MARL environment (MA-POMDP) supporting cooperative, competitive, and mixed multi-agent scenarios with continuous control, compatible with EPyMARL and MARLlib.
  • Modular reward framework exposing inter-body metrics (relative distance, probability of collision, line-of-sight) and body-specific metrics (fuel consumption, mass variation).
  • Real-time Python-based 3D visualization — claimed first in its class for orbital RL.
  • Validation against 31 Starlink satellites achieving MAPE 0.16% vs. real ephemeris data.
  • Open-source, reproducible replications of prior RL orbital missions (Kolosa [40], Herrera [31]).

Results

  • Starlink validation (31 satellites, 16.6-hour propagation): Low-RMSE group mean 24.14 m; medium-RMSE group mean 83.75 m; high-RMSE group mean 1924.90 m; 2-hour horizon residuals remain low across satellites.
  • Hohmann transfer (30 km altitude, DQN/PPO): Agent learned near-optimal strategies matching theoretical semi-major axis values; minor deviations in inclination from thrust misalignment.
  • Collision avoidance maneuver: PPO (continuous) outperformed DQN (discrete) under full realistic perturbations during evaluation despite both being trained on simplified dynamics; PPO showed greater POC reduction generalization.
  • GEO constellation coordination (4-satellite, PPO + FRL): Agents maintained larger angular separation and non-drifting altitude over 4 days; policies generalized to unseen third-body, SRP, and drag perturbations; some initial configurations failed to converge to target separation within the episode.

Limitations

  • High-RMSE satellite group (mean 1924.90 m) likely caused by missing or inaccurate physical property data (drag/reflection coefficients, shape); Bayesian tuning cannot compensate for unknown parameters.
  • Scalability remains challenging with high-fidelity numerical propagation: each body experiences continuously evolving distinct forces, making large-constellation training expensive.
  • GEO coordination policies fail to increase overall angular separation for some initial configurations within the allotted time.
  • Validation window for real-world RMSE is limited (16.6 hours); long-horizon accuracy is not demonstrated.
  • No LLM- or foundation-model-based agent integration; all experiments use classical deep RL algorithms.

Relevance to Agentic AI / LLM Agents

OrbitZoo is a domain-specific MARL benchmark rather than an LLM-agent platform, but it exemplifies the class of high-fidelity physical-world testbeds needed to evaluate autonomous decision-making agents operating under partial observability, continuous action spaces, sparse/delayed rewards, and real-world sim-to-real gaps — all core challenges for deployable agentic systems. Its CTDE architecture and modular reward design are directly transferable design patterns for multi-agent LLM orchestration research where agents must coordinate under communication constraints. The emphasis on grounded reward functions and uncertainty propagation is relevant to researchers developing physically grounded agent evaluation environments beyond text benchmarks. For those tracking agentic AI in safety-critical domains, OrbitZoo's pursuit–evasion and collision avoidance scenarios offer a well-validated physical testbed for safe exploration and robust policy learning.