Skip to main content
LivestreamHear more about how Niantic Spatial is working with NVIDIA and Flexion to close the Sim-to-Real gapWatch now
Contact sales
Company

The team, partners and resources behind Niantic Spatial.

News

Seeing the Whole Space: What 360° Capture Changes for Physical AI

In the second of our three-part series exploring how models and real-world data are helping machines understand their surroundings, we turn to 360° imagery. Anvith Ekkati explains how relatively inexpensive cameras, combined with increasingly sophisticated AI models, can create accurate representations of real environments, and why this is an important requirement for physical AI.

In the first article in this series, we looked at how localization connects what a machine sees with a wider understanding of where it is. But physical AI also needs a sufficiently rich model of the specific environment it has to operate in. A robot working in a factory does not simply need something that looks like a factory. It needs an understanding of that factory: its dimensions, surfaces, obstacles and layout.

Historically, obtaining that ground truth has involved a trade-off. Specialist surveying systems can capture environments precisely, but require expensive specialist equipment, deep expertise and a lot of time. Off-the-shelf cameras are much cheaper and easier to deploy, but turning their images into sufficiently complete and accurate spatial models is a far harder computer-vision problem.

360° cameras are helping to solve it.

A wider view, faster and for less

Capturing a space with a conventional phone camera requires someone to think about where to point it and whether enough of the environment has been recorded. A 360° camera observes the entire scene around it as someone moves through a space. Outdoors, we have captured areas as large as 10,000 square meters in around ten minutes while maintaining broad visual coverage.

That makes collecting real-world observations much easier. But a sequence of 360° images is still not a 3D model. The difficult part is extracting useful spatial understanding from it.

Niantic Spatial’s reconstruction pipeline first has to identify common features across different views and use them to work out where the camera was when each image was captured. It then has to solve another fundamental problem: metric scale. A reconstruction derived from images does not inherently know whether something is ten centimeters or ten meters wide. That might not matter for an attractive visualization. It matters enormously if a robot is expected to move through it.

There is a similar distinction between something that looks three-dimensional, and rich geometry a machine can actually use. Gaussian splats can produce remarkably convincing representations of real places, but physical AI also needs reliable underlying surfaces. Think of a video game where the graphics and the invisible boundaries are slightly out of sync: a route can look open even though the player cannot pass through it. For robot simulation, the same principle applies. What the robot sees needs to match the surfaces it can actually interact with. Our work on foundation models for depth estimation helps recover much denser geometry, including areas such as plain walls where traditional feature-based computer vision can struggle. We can combine this with the visual representation to produce an aligned collision mesh, so a simulated robot can see a photorealistic environment and interact with geometry derived from the same physical place.

This is where the significance of cheaper capture becomes clearer. The advance is not simply that a 360° camera can record a factory quickly. It is in the increasingly sophisticated models, like those we are developing here at Niantic Spatial, which can turn those inexpensive observations into metric spatial information that previously required much more specialized capture.

Knowing where you stand

The same data can also support localization. Traditional systems identify visual features in a camera image, match them against a map and use those correspondences to calculate position. They can work extremely well, but environments such as industrial facilities expose their limitations. Rows of similar pipes, valves and machinery provide fewer distinctive landmarks, while changing lighting can make the same location appear very different.

One of our latest models takes a different approach to addressing this issue. Trained on billions of posed real-world images, the model learns compact representations of places that a new camera image can be compared against to recover position and orientation. Rather than relying only on individually engineered features, it can draw on wider visual context learned across many different environments. This makes localization more robust to change. For example, the model can establish a camera's position at night using a map captured during daylight hours. In another demonstration, it successfully localized within a park that had been severely damaged by flash flooding after it was originally mapped, despite substantial changes to the environment.

This is important because reconstruction and localization are not separate ends in themselves. A 360° capture can provide both a detailed model of a real environment and a reference for establishing position within it. We can also use approximate GPS data from an ordinary camera or phone to accurately position outdoor captures in global coordinates, without specialist surveying equipment. This means a robot can understand not just where it is within a particular environment, but where it is in the wider world – a capability that is already proving valuable to our customers. Semantic information can add another layer by describing what the environment contains.

360° imagery is therefore one part of a much wider spatial picture. Ground-level capture provides rich detail about the spaces through which people and machines actually move, while aerial and satellite imagery offer very different perspectives and far greater geographical coverage. No single source provides everything. The opportunity is to develop models that can draw useful spatial understanding from these different real-world observations and connect them where needed.