At SIGGRAPH 2025, the International Conference on Computer Graphics, held in Vancouver, Canada, NVIDIA Research presented a series of software and technology innovations for "Physical AI." Several vice presidents and research leaders delivered special speeches, explaining how graphics and simulation can jointly advance the development of physical AI.
In the development of modern robotics, self-driving cars, and smart spaces, "Physical AI" has become a core driver of technological advancement. This technology combines neural graphics, synthetic data generation, physics-based simulation, reinforcement learning, and AI reasoning capabilities, enabling AI systems to not only "see" but also "understand" and interact with the real world.
For nearly 20 years, NVIDIA's research team has been deeply engaged in the intersection of AI and graphics, becoming a key force in advancing this technology.
The two-way advancement of AI and simulation
Sanja Fidler, vice president of AI research at NVIDIA, said that advances in AI are driving more powerful simulation capabilities, and more accurate simulations, in turn, are accelerating the development of AI systems. There is a strong two-way connection between the two.
In this announcement, NVIDIA announced several new tools and models for physics-based AI, including:
• NVIDIA Omniverse NuRec 3D Gaussian splatting library for large-scale world reconstruction.
• Updates to NVIDIA Metropolis visual AI platform.
• NVIDIA Cosmos and NVIDIA Nemotron reasoning models, including Cosmos Reason, a visual language model with physical reasoning and common sense judgment capabilities, enabling robots and AI agents to make human-like judgments.
Behind these achievements are multiple breakthroughs in papers from NVIDIA research teams around the world in areas such as neural rendering, real-time ray tracing, synthetic data generation, and reinforcement learning.
Building a virtual world to train real robots
The development of physical AI often begins with the creation of highly realistic, physically accurate 3D environments. Without such virtual environments, developers cannot effectively train robots in simulation, as the skills learned in virtual environments may not transfer seamlessly to the real world.
Ming-Yu Liu, Vice President of Research at NVIDIA, gave an example. An agricultural robot needs to precisely control its arm strength to pick peaches without causing damage. A manufacturing robot must assemble electronic components with micron-level precision; even the slightest error can lead to failure. These high-precision actions must first be repeatedly tried and learned in a safe and realistic virtual space.
NVIDIA's long-standing research in ray tracing and real-time graphics rendering is the core that supports these highly realistic simulations. Simultaneously, AI technology is also being used to rapidly transform photos or videos into interactive 3D virtual scenes. This "2D to 3D" reverse rendering capability significantly lowers the barrier to entry for creating virtual worlds.
New tools and research breakthroughs
During this SIGGRAPH, NVIDIA also announced a number of research results and tools:
• ViPE (Video Pose Engine):
Developed in collaboration between the Spatial Intelligence Lab and the NVIDIA Isaac team, it can infer camera motion from typical footage (such as dashcams, handheld footage, and even movie footage) and generate high-precision depth maps and 3D geometry annotations.
• Generative AI predicts future scenarios:
The Deep Imagination Research team uses computer vision, Transformer, and visual generative models to enable physical AI systems to predict possible changes in the environment, such as detecting a car running a red light or judging the risk of a cup falling on a table.
• Structurally stable 3D reconstruction:
The new research solves the problem that 2D models generated from 3D images are prone to collapse in physical simulations, ensuring that the generated objects conform to real physical structures and avoiding unreasonable physical reactions during virtual training.
• Real-life action characters:
Combining the motion generator with a physical tracking controller, highly realistic synthetic data of complex actions such as parkour is generated. This can be used to train virtual characters or robots to learn difficult actions and be applied in scenarios such as rescue or walking in harsh terrain.
• AI assistant for material detail generation:
Using a diffusion model and a differentiable physically based renderer (PBR), artists can add details such as weathering and aging to 3D objects simply by entering text descriptions, significantly reducing content production time.
• Differentiable ray occlusion query:
The new technology can reconstruct 3D geometric images from images and videos more quickly and accurately, and combined with generative basic models, it becomes an AI assistant for 3D content production.
Promoting industrial digitalization and intelligence
Through these research and technologies, NVIDIA seamlessly integrates graphics rendering, physics simulation, and generative AI, providing a complete foundation for physical AI development. These technologies not only accelerate game and virtual content creation but can also be applied to smart factories, smart cities, autonomous driving, and robotics, accelerating industrial digital transformation.
NVIDIA points out that when the virtual world is highly consistent with real physical rules, AI systems can learn more safely and efficiently in virtual space and ultimately bring their skills to the real world.
