Skip to content

GS-Agent: Creating 4D Physical Worlds With Generative Simulation

🕒 Published (v1): 2026-07-23 17:04 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GS-Agent is a multi-agent framework that converts natural language prompts into physically plausible 4D worlds (dynamic 3D scenes over time) by placing a physics engine in the loop alongside three specialized agents: a Manager, an Entity Agent, and a Render Agent. Unlike text-to-video diffusion models that generate plausible pixels, GS-Agent writes executable simulation code that produces ground-truth physics states, enabling direct kinematic extraction and autonomous error recovery. It outperforms Sora2 and Wan2.2 on physical plausibility and instruction fidelity while enabling fine-grained user control via natural language.

Problem

Generating 4D physical worlds from natural language faces two unsolved challenges: (1) learned generative models (diffusion-based video generators) produce visually coherent but physically implausible outputs because they approximate pixel distributions rather than physical laws; (2) traditional computer graphics pipelines require extensive manual tuning of materials, motions, and rendering, making them inaccessible. Existing agentic approaches (e.g., SWE-Agent) can write simulation code but lack visual feedback and physics-aware material tuning, resulting in incomplete or unstable simulations.

Method

GS-Agent decomposes 4D world creation into two top-level concerns — entity management and rendering configuration — handled by three collaborating agents that interact with the Genesis physics engine through structured tool interfaces and generate executable simulation code:

  • Manager Agent: parses the natural language description, formulates a multi-step plan, delegates one sub-task at a time to specialized agents, configures global simulation parameters (timestep, gravity, MPM/SPH solver settings), controls timeline advancement (advance/reset), and verifies each sub-task result before proceeding.
  • Entity Agent: retrieves 3D assets via CLIP-based semantic search (BlenderKit, PolyHaven) or falls back to text-to-3D generation (Meshy); places entities with AABB-based scale/orientation alignment; iteratively tunes material parameters (Young's modulus \(E\), Poisson's ratio \(\nu\), yield stress \(\sigma_y\) for MPM; particle radius for SPH) using physics feedback; and programs per-step motion control functions (PD controllers, kinematic trajectories, particle emitters).
  • Render Agent: grounds verbal camera directives in world coordinates, codes explicit camera trajectories (orbits, dollies, tracking), synchronizes frame capture with simulation timesteps, and configures lighting (directional lights, environment maps, local sources).

Agents communicate via a SendMessage tool; multimodal feedback (physics boundary checks, runtime logs, intermediate renders, videos) is used for iterative refinement. The final output is both executable simulation code and a rendered video, uniquely enabling direct extraction of 3D center-of-mass kinematics as ground-truth physical state.

Key Contributions

  • End-to-end multi-agent framework that emulates human 4D world-building workflows and automates the entire pipeline from natural language to rendered video.
  • Physics engine integration via structured tool interfaces enabling code-level interaction with rigid-body, MPM (deformable), and SPH (fluid) solvers.
  • A more rigorous evaluation metric, State-PIS, which computes the Physical Invariance Score from exact 3D kinematics extracted from the simulator rather than approximate pixel-space centroids (Video-PIS).
  • Emergent autonomous error detection and recovery (e.g., detecting a leaking 3D asset and programmatically patching it with rigid material fills) without human intervention.
  • Fine-grained natural language controllability over camera trajectories, entity spawning, and physical parameters (e.g., velocity) at runtime.

Results

Evaluated on 24 scenes from NewtonGen (12 physical laws) and 30 novel complex scenes with multi-object interactions and dynamic cameras; 15-participant user study with 270 responses:

  • Video-PIS (NewtonGen): GS-Agent 0.71 vs. Sora2 0.62, Wan2.2 0.46, SWE-Agent 0.41, SWE-Agent w/ Visual 0.49 (higher is better).
  • State-PIS (direct kinematic extraction; unavailable for video models): GS-Agent 0.83 vs. Wan2.2 0.57, Sora2 0.44.
  • Alignment Score (Perception Encoder similarity): GS-Agent 32.2 vs. Sora2 30.6, Wan2.2 29.8, SWE-Agent 25.8, SWE-Agent w/ Visual 26.8.
  • Aesthetic (VBench metric): GS-Agent 47.6 vs. Wan2.2 58.1, Sora2 48.5 (GS-Agent trails on pure aesthetics).
  • User study — Physical Plausibility: GS-Agent 4.33 vs. Sora2 4.01 (5-point Likert).
  • User study — Camera Controllability: GS-Agent 4.32 vs. Sora2 3.92.
  • User study — Content Alignment: GS-Agent 4.70 vs. Sora2 4.65.
  • User study — Aesthetics: GS-Agent 3.86 vs. Sora2 4.18 (only dimension where GS-Agent is not first).
  • Ablation confirms multi-agent design is critical: a single monolithic agent produces incomplete scenes; removing the specialized agent-simulator tool interface causes unstable simulation code generation.

Limitations

  • Aesthetic quality lags behind closed-source text-to-video models (Sora2) on both automated metrics and user ratings, reflecting the gap between physics-rendered visuals and neural video synthesis.
  • Depends on external 3D asset libraries (BlenderKit, PolyHaven) and a third-party text-to-3D service (Meshy); falls back to geometric primitives when both fail, limiting morphological fidelity.
  • Uses GPT-5 as the backbone; performance is backbone-dependent, and ablation results on backbone variation are only partially presented (paper text is truncated).
  • State-PIS metric is only computable for physics-engine outputs and cannot be used to directly compare against video-generation baselines on the same footing.
  • Evaluation scenes (54 total) are relatively small-scale; generalization to highly complex or open-domain scenes is not demonstrated.

Relevance to Harnesses / Meta-Harnesses

GS-Agent is a canonical example of a hierarchical agent harness: the Manager Agent acts as a meta-controller that plans, delegates, and verifies, while the Entity and Render Agents are specialist sub-agents with scoped tool access — a direct instantiation of the harness pattern where a coordinator orchestrates downstream workers through structured interfaces. The system's use of a SendMessage tool for inter-agent communication and iterative task reassignment based on verification outcomes closely mirrors meta-harness designs that emphasize feedback-driven refinement loops. The autonomous error recovery capability (detecting simulation failures, generating a fix, and re-verifying) demonstrates how a meta-harness can achieve robustness beyond what individual agents can provide. For researchers tracking harness architectures, this paper is notable for grounding the harness in a physics engine rather than a code interpreter or web browser, showing that the meta-harness pattern generalizes to simulation environments with structured state feedback.