In this series of posts, we explore how people and machines can understand the places around them, and the combination of models and real-world data that makes that possible. We start with why localization needs more than GPS and pre-scanned maps, then look at 360° imagery and, finally, aerial data captured by drones and satellites, exploring how these can contribute to a richer understanding of real-world environments.
A delivery robot can identify every obstacle in front of it, but after a downtown battery swap, a three block GPS error means it might not know which street it’s on. Knowing what’s in view does not establish which street it occupies or which direction it faces. As ambitions for physical AI expand, machines need to connect what they see with the particular places where they must act, relating their immediate surroundings to a wider geographic understanding of the environment. That is why localization, the process of determining where a person or machine is within an environment and in which direction it is facing, is so important for the successful application of physical AI.
Harder than it looks
Robots can do some of this already. Local Simultaneous localization and Mapping (SLAM) and odometry can track movement within a robot’s own coordinate frame, but relating that frame to a geographical location requires another reference. That reference can be obtained in a number of ways. GPS is one, although in dense urban environments its errors can leave a robot uncertain of even the simplest details, such as which side of the road it occupies.
An alternative to GPS is to scan the location, creating a detailed map against which a robot can match its camera observations. Capturing this data with phones or 360° cameras works well and can enable centimeter-level localization. But it means each new operating area must be scanned before localization within that same area is possible.
This is not always an option. Scanning an entire city to enable localization for robots that deliver goods to homes and shops is often uneconomical. In some harsh environments, scanning an area in advance may simply not be possible. That is why visual localization needs to work beyond individually scanned sites.
Satellite imagery and digital vector maps offer that wider coverage, but interpreting them alongside a ground-level camera view is difficult. A camera on a robot sees a building’s façade; while a satellite sees its roof with its facades barely visible or hidden entirely. Sensors, capture angles and resolutions differ, while shadows, cloud cover, vegetation and temporary objects such as scaffolding or market stalls alter or obscure the scene. Vector maps introduce another gap: roads and buildings appear as geometric shapes rather than photographs.
The available evidence becomes limited further at pavement or sidewalk level. For example, delivery robots share the pavement with pedestrians whose paths and behaviour add complexity. Low-mounted cameras may face a featureless wall on one side and parked cars on the other, with little of the wider scene visible. This means that any position estimate must come with reliable uncertainty information, allowing the receiving system to judge how much to trust it when visual evidence is weak.
Model answer
This is why Niantic Spatial is developing pioneering new models that can interpret these different representations and relate them to visual cues, connecting observations that may have little visual resemblance.
Our localization service puts these capabilities into practice. Given camera images and a coarse GPS estimate, even one several blocks off, it retrieves the relevant maps and returns candidate positions and headings, with well-calibrated uncertainties. We also fuse existing sensor information from the robot’s on-board SLAM, Inertial Measurement Units (IMUs), or even just its wheel encoders, to produce accurate motion trajectories that are robust to noisy odometry such as wheel slip during periods of snow and rain. Using existing maps also makes this approach practical to deploy at city scale. Localising against overhead imagery and digital maps allows operators to extend their coverage without commissioning a dedicated scan of every area they serve.
Delivering results
Our work with companies designing and implementing urban delivery robots, shows this approach in practice. After a robot has been transported or restarted following a battery reset, it needs to establish its location and orientation. GPS readings can be several blocks adrift, making even the robot’s street uncertain. By matching its camera views to existing geospatial map data we help resolve that ambiguity and provide robots with a firmer basis for deciding where to move.
We use views from four cameras to widen the available field of view, while training on additional Niantic Spatial data and improving our training methodology helped us improve robustness to the difficult perspectives and conditions these robots face. These conditions also make uncertainty part of the engineering problem. We have worked to calibrate reported uncertainty against localization errors, giving developers information with which to judge how much to rely on the returned positions.
The immediate benefit is a more dependable position estimate when GPS and the ground view provide limited evidence. The broader future opportunity depends on connecting localization with complementary capabilities such as reconstruction, which supplies geometry, and semantic understanding, which describes what is present. Together, these could let a robot relate its current view to both the structure and meaning of its surroundings or connect a worker’s camera to information about the facility they are inspecting. Our current localization API returns position and orientation estimates; this richer, integrated workflow is something we’re working towards.
The next two posts in this series will explore how 360° capture and aerial data contribute different perspectives and coverage to that wider effort. Across all three, the technical challenge is to turn diverse observations of actual places into models that can interpret the physical world.