How to Use AI Agents to Prepare 3D Scenes for Simulation

NVIDIA Details Agentic Workflow to Automate Simulation Scene Prep

NVIDIA has outlined a multi-agent workflow that automates the complex and often laborious process of preparing 3D scenes for robotics simulation. The system uses a high-level coordinating agent, such as OpenAI's Codex or Anthropic's Claude, to manage a team of specialized subagents that use NVIDIA Omniverse Libraries to make a standard Blender scene simulation-ready. This approach addresses a critical bottleneck in robotics development, where the manual effort required to add physics, semantic labels, and other metadata can stall projects before training in simulators like Isaac Sim can even begin.

The technical architecture relies on a clear division of labor between different AI components, with OpenUSD serving as the foundational data layer. The process begins with an agent inspecting a Blender scene via a Model Context Protocol (MCP) server to inventory its contents. Based on this inventory, the main orchestrator deploys specialized subagents built with NVIDIA NemoClaw to perform discrete tasks. These agents use specific Omniverse Libraries as tools to act on the scene.

• Orchestration: A general agent like Codex interprets the user's goal and plans the overall task.
• Subagent Deployment: NVIDIA NemoClaw manages specialized agents that reason and execute specific jobs.
• Tool Execution: Subagents call Omniverse Libraries like ovphysx for physics authoring and ovrtx for preflight rendering.
• Validation: The final scene is checked against SimReady validation profiles to ensure compatibility with target simulators.

By automating scene preparation, this workflow stands to significantly reduce the time and specialized knowledge required to create viable digital twins for physical AI systems. The impact extends beyond simple efficiency gains; it makes high-fidelity simulation more accessible, potentially accelerating the entire research-to-deployment pipeline for robotics. This model also serves as a practical blueprint for how general-purpose LLMs can orchestrate teams of specialized, tool-using agents to solve complex, domain-specific engineering problems, moving agentic AI from conversational theory to tangible industrial application.

This workflow demonstrates a mature pattern for agentic AI, moving beyond conversational assistants to structured, multi-agent systems that solve domain-specific engineering bottlenecks. By using OpenUSD as a common data fabric and Omniverse libraries as callable tools, NVIDIA is positioning its platform as the essential middleware for building and training physical AI.