Understand
Geo-referenced spatial intelligence at the pixel level, combining segmentation and contextual understanding that will give humans, robots, and language models and AI agents real-time awareness for smarter and more efficient decision-making.
Built on open-set semantics, Niantic Spatial Understand encodes each 3D point with meaning, enabling context-rich, queryable 3D maps interpretable by humans and machines.
Semantic Understanding
Building Next-Generation Semantics
Per-point semantic understanding: Each spatial point encodes an open, high-dimensional descriptor capturing geometry, material, and contextual features.
Open-set generalization: Identifies and adapts to new concepts without retraining or relying on fixed category taxonomies.
Cross-modal grounding: Connects visual, spatial, and linguistic information within a unified semantic framework.
Continuous semantics: Delivers nuanced understanding of relationships across objects, surfaces, and environments for richer spatial reasoning.
Niantic Spatial Understand in Action
What Understand Enables
Queryable maps: Ask open-ended spatial questions like “Where does vegetation encroach on power lines or access roads?”, “What terrain is traversable for inspection vehicles or drones?”, and “Which structures provide shelter or shade for field operations?”
Context-aware autonomy: Robots and AI agents reason about their surroundings using meaning, not just geometry.
Adaptive intelligence: Maps evolve with new data, improving recognition and inference over time.
Built for integration: Designed for multi-modal queries across enterprise systems, Physical AI agents, and in-field robotics workflows.
Built for the Real World
Defense
Real-time scene interpretation, situational awareness, and mission-critical reasoning for safer, more informed operations.
Robotics & Autonomy
Semantic navigation, task recognition, and adaptive decision-making powered by spatial understanding.
Intelligent Field Operations
Automates inspection and asset intelligence for oil and gas, utilities, and large-scale infrastructure environments.
Frequently Asked Questions
What is semantic understanding in 3D?
+
3D semantic understanding adds meaning and context to a reconstructed environment, not just geometry. Niantic Spatial Understand encodes each 3D point with information about features such as geometry, materials, and context, creating maps that humans, machines, and AI agents can query, measure, and reason over.
What are open-set semantics?
+
Open-set semantics allow a system to identify and reason about concepts beyond a fixed list of predefined categories. In Understand, open-set generalization is designed to adapt to new concepts without retraining for every new category or relying only on a closed taxonomy. That makes the semantic layer better suited to open-ended questions about real environments, including objects, materials, relationships, and conditions.
What is per-point semantic understanding?
+
Per-point semantic understanding attaches a semantic descriptor to each point in a 3D scene. The descriptor can capture geometric, material, and contextual features, allowing the system to search or compare specific parts of an environment rather than treating the entire scene as one label. This creates a spatially grounded foundation for segmentation, measurement, natural-language search, and downstream AI workflows.
Can you query a 3D map using natural language?
+
Yes. It is designed to support open-ended, multimodal questions about a 3D environment, such as where vegetation encroaches on infrastructure, which terrain is traversable, or which structures provide shelter or shade. Cross-modal grounding connects visual, spatial, and linguistic information so the answer can be tied to a location or region in the mapped environment.
How is 3D semantic understanding different from traditional object detection?
+
Traditional object detection or closed-set segmentation generally assigns predefined labels to visible regions. 3D semantic understanding adds spatial grounding and context: it can represent meaning at individual 3D points, connect relationships across objects and surfaces, and support open-ended queries. Understand is positioned as a layer for interpreting and operating in a measured 3D environment, not simply labeling isolated 2D images.
What can 3D semantic understanding be used for?
+
3D semantic understanding can support scene interpretation, situational awareness, semantic navigation, task recognition, inspection, and asset intelligence. Humans, robots, and AI agents in can use meaning, not geometry alone, to interpret surroundings, identify relevant areas, reason about conditions, and make more informed decisions.
How does spatial semantics help AI agents and robots?
+
Spatial semantics gives AI agents and robots a shared, queryable representation of the physical world. By grounding visual, spatial, and linguistic information in a 3D map, it helps machines interpret their surroundings and act on context, not just geometry. That foundation supports applications such as semantic navigation, task recognition, adaptive decision-making, inspection, and asset intelligence.