NVIDIA Unveils Largest Indoor Synthetic Dataset at CVPR for Advancing Physical AI

NVIDIA has made a significant contribution to the annual AI City Challenge at the Computer Vision and Pattern Recognition (CVPR) conference by providing the largest ever indoor synthetic dataset. This effort aims to aid researchers and developers in creating AI solutions for smart cities and industrial automation, according to the NVIDIA Blog.

Advancing AI for Smart Cities and Industrial Automation

The AI City Challenge, which attracted over 700 teams from nearly 50 countries, focuses on developing AI models to enhance operational efficiency in various physical settings, including retail and warehouse environments, as well as intelligent traffic systems. This year's challenge featured datasets generated using NVIDIA Omniverse, a platform that supports the creation of Universal Scene Description (OpenUSD)-based applications and workflows.

Creating and Simulating Digital Twins

Large indoor spaces, such as factories and warehouses, require solutions that can observe and measure activities, optimize operational efficiency, and ensure human safety. To meet these needs, researchers are leveraging computer vision models trained on large, ground-truth datasets for a variety of real-world scenarios. However, collecting such data is often challenging, time-consuming, and costly.

Physically based simulations, such as digital twins of the physical world, are increasingly being used to enhance AI simulation and training. These virtual environments help generate synthetic data for training AI models, allowing researchers to run numerous “what-if” scenarios in a controlled setting while addressing privacy and AI bias issues.

Building Synthetic Datasets for the AI City Challenge

This year, NVIDIA contributed datasets for the Multi-Camera Person Tracking track, which saw the highest participation with over 400 teams. The track used a benchmark and the largest synthetic dataset of its kind, comprising 212 hours of 1080p videos at 30 frames per second across 90 scenes in six virtual environments, including a warehouse, retail store, and hospital.

These scenes, created in Omniverse, simulated nearly 1,000 cameras and featured around 2,500 digital human characters. The benchmarks were produced using Omniverse Replicator in NVIDIA Isaac Sim, an application that enables developers to design, simulate, and train AI for robots and autonomous machines in virtual environments.

Driving the Future of Generative Physical AI

Researchers and companies worldwide are developing infrastructure automation and robots powered by physical AI, which can autonomously perform complex tasks. Generative physical AI uses reinforcement learning in simulated environments, where it perceives the world using simulated sensors and performs actions grounded by physical laws.

NVIDIA is further advancing simulations with the recently announced Omniverse Cloud Sensor RTX microservices. These microservices enable physically accurate sensor simulation, accelerating the development of fully autonomous machines by allowing large-scale tests in virtual environments, significantly reducing the time and cost associated with real-world testing.

Showcasing Advanced AI With Research

Participants in the AI City Challenge submitted research papers, some of which achieved top rankings. All accepted papers will be presented at the AI City Challenge 2024 Workshop on June 17. NVIDIA Research will also present over 50 papers at CVPR 2024, showcasing breakthroughs in generative physical AI with potential applications in autonomous vehicle development and robotics.

For more information, read about NVIDIA Research at CVPR and learn more about the AI City Challenge.