State-of-the-Art Geometry Your Policy Learns From | Niantic Spatial, Inc.

For autonomy teams, reconstruction quality is not only a visual problem. It is a problem of physics, control, and trust.

A robot policy can only learn from the environment represented in its training loop. If the collision mesh smooths away a real obstacle, invents a bump that is not there, or misplaces a surface relative to the rest of the scene, the policy is learning from the wrong world. The resulting behavior may look convincing in simulation and still fail when it meets reality.

That is why the geometry underneath a reconstruction matters as much as the images it can reproduce. A scene becomes useful for autonomy when its surfaces, boundaries, and spatial relationships remain coherent beyond the captured views, and when a policy can train against them with confidence.

Photorealistic rendering is not the same as reliable geometry

A splat optimized mainly to reproduce captured views can find multiple explanations for the same images: a wall may look flat from the source cameras while remaining warped underneath, a thin structure may drift from a new viewpoint, and a reflective or transparent surface may appear plausible without corresponding to a stable physical surface.

Those errors often remain hidden in a single rendered view. They become consequential when the representation is used to generate a mesh, render a novel viewpoint, or train a policy that must interact with the scene. Splats can look excellent. However, appearance alone isn’t able to pin down the geometry of the scene between captured views.

Our approach makes geometric consistency part of the optimization process. For each view, the system establishes where surfaces should be in depth. During splat optimization, it renders the splat’s depth and compares it against that evidence, encouraging the Gaussians to occupy the right locations and the surfaces to remain stable as the viewpoint changes. Where additional scene geometry is available, it can also be incorporated as a constraint, strengthening the representation further.

This also explains why simply training the model for longer won’t solve the problem. Without a geometric constraint, more optimization can overfit the Gaussians to the captured images, causing the geometry between those views to deteriorate.

The result is a single representation that delivers both photorealistic rendering and reliable geometry. In our comparisons, the difference is especially visible on large planar surfaces and in low-texture regions: walls stay flatter, floors remain more stable, and extracted surfaces contain less of the noise that appearance-only optimization can introduce. For autonomy, the geometry has to match the site the robot will actually work in.

A capability built into a production system, not delivered by a single model

This capability builds on MVSAnywhere, our published zero-shot multi-view depth model, which many teams have used as a foundation. But what we run in production today goes well beyond that paper: it combines geometry regularization during splat optimization with the engineering required to preserve quality and generalize across large, varied scenes and capture conditions. The result is a photorealistic representation with a stronger geometric foundation, delivering consistent imagery and reliable structure underneath.

The difference lies in the full system: how geometric information is incorporated, how existing scene evidence is used, how splats are optimized, and how the pipeline is tuned for large and varied environments. A demo can show that a scene looks convincing. A production reconstruction system has to produce geometry that remains coherent from novel viewpoints and across the scenes, camera configurations, and operating conditions customers actually encounter.

That is the standard that matters for downstream autonomy. The output must be useful not only as a render, but also as a mesh, a collision surface, and training data for systems that have to operate in the physical world.

Better geometry from imperfect capture

Geometry foundation models and regularization make the quality of the initial capture less decisive. Instead of asking teams to repair every artifact by hand, we use geometric evidence to constrain the representation and improve the structure underneath. That lets teams do more with the data they already have, including limited-coverage input, while maintaining a high bar for the resulting geometry.

The same holds at scale. Through engineering and tuning across the whole system, we’ve taken the pipeline from room-sized scans to large facilities without lowering the quality bar. Transparent and reflective surfaces remain difficult because their appearance does not map cleanly to one physical surface, so they require specialized treatment and careful handling. The objective is not to make imperfect capture look acceptable from one view; it is to produce geometry that remains useful downstream.

Why teams choose not to build this themselves

Many autonomy teams can assemble a reconstruction pipeline, and some have built theirs on our published research. They can produce a scene that looks convincing, but the harder problem is turning those pieces into a dependable system that works across real scenes and capture conditions.

That dependability is what keeps teams from having to lay flat planes over their floors and walls by hand to cover bumps that aren’t there. It requires geometry foundation models, regularization, optimization, evaluation, edge-case handling, and the engineering to make them work together. It also requires understanding where visual fit diverges from spatial truth, how errors compound during mesh extraction or policy training, and how to maintain quality as the scene and capture setup change.

That’s why teams work with us. They get geometry they can train against without having to reproduce years of research and systems work in-house. And because that foundation is trustworthy, they can immediately benefit from new capabilities we’re building on top of it, including scene variation and relighting.

Scene from our Embodied AI Places Library

Train policies on geometry they can trust

A mesh that accurately captures real planes, edges, clearances, and obstacles gives a policy a more faithful environment to learn from. A mesh that encodes reconstruction artifacts does the opposite: it teaches the policy to compensate for errors that do not exist in the physical world. Those errors may surface later as brittle behavior, unexpected contacts, or a wider gap between simulation and deployment.

For that reason, the collision mesh should not be treated as a byproduct to repair after reconstruction. Reliable geometry needs to be established upstream, as part of the representation and optimization process. The benefit is a scene that does more than look real: it gives autonomy systems the spatial structure they need to reason about the physical world.

This work is still active. We’re continuing to improve reconstruction from limited coverage, including from the cameras robots already carry.

The standard we are building toward is not a scene that merely looks right, but geometry that supports reliable transfer from simulation to reality.