Ground Truth: Putting the "World" in World Models
Ground Truth: Putting the “World” in World Models
“LLMs gave AI fluency with language. Vision models gave it the ability to interpret images. World models are giving it physics intuition. What physical AI still largely lacks is a grounded, accurate, continuously updated understanding of specific real-world spaces.”
For more on Niantic Spatial's products for embodied AI, see here.
The next frontier of AI is the physical world. At the center of recent exciting developments in robotics are so-called world models. These models try to capture the state of the world and predict how the world reacts to actions taken in it.
Their main promise lies in planning and simulation. In planning, world models allow a robot to reason about the sequence of steps (actions) that is necessary to reach a goal or to fulfill a task. In simulation, world models allow training of robots on a massive scale and without the danger that failure results in hardware damage.
The problem with current world models: They describe and portray plausible worlds, but not the world. Not our world.
In this piece, we argue that current world models lack grounding in the real world. This lack of grounding will ultimately prevent them from reaching their full potential for physical AI. For LLMs, grounding via RAG helped to reduce hallucination. Grounding in physical space is harder but will help robot action models to select appropriate actions for a specific location. Grounded simulation will allow for evaluation and training of robotic policies in their actual deployment sites, and location-specific feedback loops will help to improve grounded simulations when deployed systems feed real-world location experience back into post-deployment training. We are building a spatial intelligence layer which offers location-specific grounding for world models at a cost that enables scale.
The disconnect of generative models
World models have a strong rooting in generative modeling. The relation becomes clear when reflecting on generative video models. Conditioned on one or more initial frames, which might themselves be generated, these models predict the next set of frames. These models can be trained on internet-scale video databases and reach astonishing visual fidelity.
Through the act of generation, video models answer the question “What happens next?” What is missing for a world model is the causal relationship to actions: “What happens next if I do X?”
There are two main data sources available to address this gap, both with advantages and disadvantages, but both with a fundamental limitation.
First, one can utilize the same video data corpus as before, where actions, i.e. what actually is happening in each video, have been attached post-hoc through manual labelling or (semi-)automatic annotation. Or, second, one resorts to renderings of synthetic environments where ground truth actions are available, for example from game engines. The tradeoff between the two lies in the reliability of the action labels (for real videos) versus the realism of the visuals and correctness of physics (for synthetic renderings).
Both sources are useful in their own right, but there is a drawback both data sources share: They are detached from spatiotemporal reality, without a grounding layer.
This is self-evident for synthetic data. These are not real places. Recordings from video games often depict fantasy worlds with their own sets of rules which can be entirely different from the rules on planet earth. Level designers will rarely consult with structural engineers on whether their creations would endure. Any ambition regarding realism must submit to the constraints of real-time computation.
Interestingly, a similar flaw holds for generic video collections. These are real places but they are still detached from physical space.
Firstly, video data fails to capture metric scale – how big things are. Models trained on video data generate scenes in arbitrary units, not meters. There are ways to infer metric scale from images and videos using visual cues, like the common size of a bench or a car that might be visible somewhere in the background. But such scale estimates are only approximate, all the while people (and likely robots) trip if the height of a step or threshold is off by a centimeter. Approximate scale estimates also break down if visual cues are simply not available. Consider an industrial site. Some of these look completely alien, even to human eyes. Is the valve in view 5cm or 50cm in diameter? There is no way to tell from an image, and training on even millions of hours of travel logs will not change that.
A Gaussian Splat of an oil refinery, where accurate geometry and metric scale matters most for physical AI
Secondly, generic video collections are not geo-located – individual recordings lose their spatial associations. Are two recordings showing the same place at different points in time? Or are they showing different places that are nearby? Or on different continents? This matters because places change in specific ways over time, and human environments were shaped by designers, architects and city planners according to their own social background with rich diversity around the globe and between industries about how those places, big and small, “work”. World generators trained on generic video collections run the risk of creating Escheresque worlds. Locations with nonsensical layouts. Houses with five staircases but no bath room. Buildings without an entrance. Footpaths that lead nowhere, just circling back into themselves. These are illustrative examples. The real disconnect can be subtle, though; for example when models fail to represent the inherently complex logic of an industrial site that was underrepresented, if at all present, in its training data.
Generated worlds have shown some promise as simulation environments for robotics. But there is a ceiling coming. And it has to do with the difference between a world that looks real and a world that is real. Robots trained in generated worlds without grounding will struggle to develop spatial intuition that helps us humans navigate new places in familiar areas without thinking too much.
Grounding via maps
One solution to the aforementioned problems has been known for a while: Maps. Unfortunately, they have shortcomings of their own.
Maps represent locations in scale-metric units and allow for measurement while a generic reconstruction does not. Since maps are associated with real places, they can be geo-spatially anchored, i.e. they refer to a specific position on earth where change can be observed over time.
Mapping is in our company's DNA. When some of us were building Google Maps, we started by licensing data from existing providers. It didn't take long to realize the maps didn't match reality. The solution was Street View – driving the roads ourselves to get “ground truth.” The lesson was simple: at some point, there is no substitute for going and looking.
Some early successes of embodied AI were possible because of accurate mapping: HD maps allowed the first autonomous cars to operate in prototype areas (and still do today). However, HD maps fell out of favour in parts of the industry not because they did not work but because of their economics.
High-fidelity mapping of a single deployment environment typically costs tens of thousands of dollars and takes weeks. Any environment change required careful re-calibration of the map and any semantic labeling was largely manual. Contrast this with existing generic video corpuses for building world models: They are available in vast amounts – a key attribute for large scale machine learning and the driving force behind any recent progress in robotics.
Gaussian splatting and cheap 360 cameras have changed the capture economics significantly. A team can walk a factory floor for ten minutes and produce the raw material for a high-quality metric-scale reconstruction in hours. That removes the access barrier to ground truth geometry.
At Niantic Spatial, we are exploring the value of ground truth spatial data in collaboration with Flexion and Nvidia. Our reconstructions let Flexion train, test and evaluate robotics in simulations of the specific site where they will be deployed, before a robot ever sets foot there. It’s still early, but policies trained in a reconstruction of the real site outperformed the same policies trained in synthetic and procedurally generated environments, and matched or beat the conventional depth baseline. For more details, see here.
There are other benefits to grounded simulation environments. Embodied AI won’t always operate in consistent, clean places; it will operate in the messiness and randomness of the world. Grounded simulation offers the potential of endless variations of lighting, objects placements, door positions, etc., but all against the same faithful backdrop of a real place as the substrate for domain randomization.
Moving beyond maps
HD Maps do not work for physical AI. They are too expensive, they go stale, they do not transfer between deployments. One response in the industry is to reject maps entirely and with it any notion of grounding in physical space. But what if we could fulfill the promise of HD maps with cheap captures? Advanced use cases in embodied AI and other fields demand precise 3D geometry, accurate representation of material properties, and semantic understanding of every part of an environment. A robot navigating a real space needs to know that this surface is carpet and that one is polished concrete. It needs to know that a door is a door and that a threshold exists even when it isn't visible in the current frame.
These requirements regarding precision and richness create a tension. Grounding is valuable because it is specific. Our collaboration with Flexion shows that. But specificity doesn't scale easily. The research direction that interests us is whether we can resolve that tension.
LLMs demonstrated the ability to generalize to situations the model has never explicitly seen. The bet we're making is that something analogous is possible for spatial intelligence: a foundation model that's capable of transferring knowledge across environments, filling in gaps in sparse reconstructions, and making the per-site ground truth problem tractable at scale. A single capture of a deployment site might be insufficient to build an HD map. But combined with prior knowledge about other sites from the same enterprise, the same industry or just a foundational understanding of physical space, grounding in a specific location becomes possible.
This is what we are building toward with our Large Geospatial Model, built on top of billions of real world data points and years of top-tier research in machine learning and computer vision. The goal is to provide grounding in physical space in the form of a spatial intelligence layer that reconciles any visual observation, sensor measurement or enterprise-specific data items in a coherent and queryable spatial representation. The Large Geospatial Model forms the technological core by offering visual positioning, semantic analysis, scene completion, etc. in a way that generalizes across industries and cultures around the globe, and across points in time.
What this means for physical AI
The potential for physical AI is significant. Cheap captures unlock evaluation and fine-tuning of policies per deployment site, across all sites of an enterprise or specialized to an entire industry. Any post-deployment experience can be fed back to the spatial intelligence layer, be it motors reporting the power it took to cross a threshold, observations about variability in lighting or measurements of how the environment reacted to manipulation. Coverage grows as robots work, not only when new sites are captured. And grounding comes to mean more than geometry and appearance. All this feedback is spatially aligned and improves post-deployment training.
John Hanke wrote earlier this year that to justify the scale of investment going into AI, you have to believe it can transform not just the 20% of the economy that is online, but the 80% that is not -- in factories, logistics networks, construction sites, and infrastructure that constitute most of the physical world. For that to happen, we need a grounded, accurate, continuously updated understanding of specific real-world spaces.