PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; PhyScensis: physics-augmented LLM agents for interactive 3D scene generation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PhyScensis is an LLM agent framework that generates physically plausible 3D scene arrangements by coupling an LLM predicate generator with a physics-engine solver and a multi-modal feedback loop. It targets robotic simulation data generation, specifically the under-explored challenge of dense, physically interacting object layouts (stacking, containment, support). It outperforms prior open-vocabulary scene generation baselines on visual quality, semantic accuracy, and physical stability metrics.

Problem

Existing LLM-based 3D scene generation methods either (a) use 2D bounding-box collision avoidance that fails in densely packed 3D configurations, (b) rely on image inpainting priors that introduce occlusion artifacts and penetrations, or (c) lack any feedback/self-correction loop, making them unable to scale to high object-count scenes. None adequately model physical predicates (stacking, containment, support) or object physical properties (mass, friction, center-of-mass), which are essential for generating realistic robotic manipulation training environments.

Method

PhyScensis operates as a three-stage agentic loop:

1. LLM Agent (Predicate Generation): Given a natural-language scene description, the LLM proposes a list of objects with annotated spatial predicates (2D positional, alignment, rotation, symmetry, grouping) and physical predicates (place-in, place-on, place-anywhere), along with object descriptions used for asset retrieval from a BlenderKit-annotated dataset.

2. Solver: A spatial solver resolves 2D positional predicates via convex-hull overlap penalties, iteratively optimizing one predicate parameter at a time. A physical solver then handles 3D stacking/containment: place-in uses physics-based packing (drop-and-settle); place-on/place-anywhere use an occupancy-grid heuristic to identify penetration-free, center-of-mass-supported candidate positions, scored against predicate parameters, then validated by a physics simulator. Probabilistic programming estimates placement stability by sampling perturbations (position, Euler angles, mass, friction, CoM shift) from a normal distribution and computing, via Bayesian estimation, the probability that perturbed states remain stable.

3. Feedback System: Reports grammar errors (ill-formed predicates), solver failures (penetrations, out-of-bounds, unsolvable stacking), and on success returns stability scores, VQA visual-quality scores, and heuristic metrics (surface coverage, compactness, object count). The agent uses this to iteratively refine predicates.

Key Contributions

• PhyScensis: an agentic LLM framework using procedural spatial+physical predicates to generate interactive 3D scenes with a comprehensive multi-modal feedback loop.
• Integration of a physics engine directly into the arrangement pipeline, enabling accurate stacking, containment, and support resolution with real object physical properties.
• Probabilistic programming stability estimator that allows controlled generation of both stable and intentionally unstable configurations.
• Demonstration of downstream utility: policies trained on PhyScensis-generated scenes transfer to unseen human-designed environments.

Results

• VQA Score (higher = better visual-semantic match): PhyScensis \(0.704 \pm 0.425\) vs. 3D-Generalist \(0.578 \pm 0.399\) vs. Architect \(0.493 \pm 0.392\).
• GPT Ranking (lower = better): PhyScensis \(1.429 \pm 0.562\) vs. 3D-Generalist \(1.946 \pm 0.731\) vs. Architect \(2.607 \pm 0.673\).
• Settle Distance (lower = more physically stable): PhyScensis \(0.003 \pm 0.008\) vs. 3D-Generalist \(0.033 \pm 0.048\) vs. Architect \(0.405 \pm 0.471\).
• Robotic task success (Reaching / Placing): PhyScensis 9/10 reaching, 3/10 placing vs. 3D-Generalist 4/10, 1/10 vs. Architect 3/10, 0/10.
• Ablation: removing the feedback system increases retry times from \(1.04\) to \(1.69\) and time-cost from \(106.41\)s to \(132.29\)s.
• Ablation on solvers (Table 3): LLM-Only (no physics solver) degrades settle distance to \(0.154 \pm 0.133\); random placement degrades VQA to \(0.415\).

Limitations

• The occupancy-grid heuristic for stacking is limited by grid resolution and lacks continuous physics during candidate search (physics only used for validation, not search).
• Asset retrieval depends on the BlenderKit dataset; coverage gaps trigger a text-to-3D fallback whose quality is not systematically evaluated.
• The probabilistic stability estimator samples from a normal distribution around the current state, which may not capture all failure modes for highly irregular geometries.
• No evaluation of end-to-end runtime at scale; feedback loops add latency (average \({\sim}106\)s per scene).
• Open-vocabulary generation is constrained by the LLM's predicate-generation quality; the ablation shows LLM-only (without solvers) produces physically unstable scenes.

Relevance to Agentic AI / LLM Agents

PhyScensis exemplifies a tightly coupled tool-augmented LLM agent pattern: the LLM acts as a high-level planner generating structured symbolic predicates, while domain-specific solvers (physics engine, probabilistic programming) serve as grounded tools that enforce real-world constraints beyond what the LLM can reason about alone. The iterative predicate-solve-feedback loop is a concrete instance of the self-correction/reflection paradigm central to modern LLM agent research. It is particularly relevant as a case study in grounding agentic outputs in physical simulators, a direction gaining traction for embodied AI and robotics data generation pipelines. The multi-modal feedback design (grammar, failure diagnosis, VQA, stability) also offers a reusable template for feedback-driven agent refinement in other tool-use domains.