Gaussian splats reproduce real places with remarkable visual fidelity, but their lighting is baked into the capture. The sun, atmosphere, shadows, and weather are recorded as static appearance, not as editable scene parameters.

Real places shouldn’t have to represent just one condition. We’re launching a relighting capability in beta for selected design partners. Our work explores how an accompanying mesh can provide the geometric structure needed to simulate new lighting conditions and transfer those changes back onto the splats.

The larger goal is to turn high-fidelity captures into configurable environments that provide controlled variation for simulation and embodied AI. This is an early, collaborative capability, not a self-service API. It is intended for technical teams that want to test scene variation in real-world environments and help shape the next stage of the workflow.

A proven technique, applied to a new pipeline

The core technique behind this work is not new. Precomputed radiance transfer has been part of computer graphics research for decades, and the 2024 PRTGS paper showed how related methods could be applied to Gaussian splats to capture soft shadows, indirect illumination, and interreflections.

The new step is applying that idea to Niantic Spatial’s real-to-sim pipeline. Across our relit variants, the underlying geometry remains unchanged; we recompute only the per-Gaussian spherical-harmonics appearance coefficients needed for the new lighting. Collision meshes, navigation meshes, and semantic labels therefore carry over untouched from one variant to the next. This preserves the same spatial and semantic foundation while varying illumination – a continuity that new captures cannot guarantee.

PRTGS provides useful research precedent for applying precomputed radiance transfer to Gaussian splats. Our workflow is optimized for generating high-quality relit variants offline in batches, with the goal of preserving the scene’s geometry while producing new lighting conditions.

The bridge between geometry and appearance

A Gaussian splat represents a scene as millions of colored 3D primitives optimized from captured imagery. That makes it highly effective for rendering a place as it appeared during capture.

But splats do not inherently contain an editable lighting model. For simulation and embodied AI, this creates a practical constraint. A team that wants the same environment at dawn, under a different sky, or after a weather change may otherwise need to recapture and reconstruct the scene.

Relighting can be approached in several ways. In our workflow, the accompanying mesh provides the geometry for simulating new illumination, which we then transfer onto the splats.

Our approach uses the aligned .glb mesh that accompanies Niantic Spatial reconstructions. Rather than treating the mesh only as collision geometry, the workflow uses it as the basis for lighting simulation.

The process is:

1. Start with a Gaussian splat and its accompanying mesh.

2. Render physically based sky lighting onto the mesh, including the sun and atmosphere.

3. Use precomputed radiance transfer to bake lighting information such as shadows and ambient occlusion.

4. Transfer the resulting lighting data from the mesh onto the individual Gaussian splats.

5. Produce relit variants of the original scene.

The mesh provides structure, while the splat preserves the visual richness.

What worked and what still needs improvement

We’ve generated variants with:

• Sun and atmosphere changes rendered through the mesh

• Time-of-day variations, including dawn-like lighting

• Atmospheric changes such as fog and humidity-driven dimming

• Rain and weather-oriented visual variants

• Wet ground, reflective puddles, and changing surface appearance

The early testing suggests three practical conclusions:

• Relighting is only as good as the underlying mesh.

• Natural visual complexity can mask some artifacts, but it does not eliminate them.

• Hard-surface interiors, glass, windows, and sharp geometry require additional validation.

Relighting builds on reconstruction quality. The accompanying mesh provides the geometry needed to simulate surfaces, occlusion, shadows, and illumination. When that geometry is inaccurate, the resulting errors can appear as visible artifacts in the relit splat.

Early tests show that natural scenes often produce the most convincing results because irregular terrain and vegetation can make small inaccuracies less perceptible. Flat surfaces, sharp corners, glass, and windows are less forgiving; indoor scenes may require additional validation.

For design partners, the current beta is an opportunity to explore what relighting can make possible today while helping us improve the workflow over time. Results will vary by scene, and we’re continuing to invest in mesh quality, rendering, and broader scene support. Our early testing shows that strong source captures and accurate aligned meshes provide the best foundation for producing convincing relit variants.

The current beta focuses on relighting a scene’s appearance and illumination. It can introduce visual changes associated with conditions such as humidity, while effects like volumetric fog and animated rain can be added as complementary simulation layers.

A collaborative beta for real-to-sim design partners

We’re starting with collaborative design-partner engagements so we can test these capabilities across representative scenes, learn where each approach works best, and continue expanding the workflow.

Each design-partner engagement will be collaborative. Together, we’ll select a representative scene and target conditions, then Niantic Spatial will review the source splat and mesh and produce an initial set of relit variants. Partners can evaluate those variants in their own applications or simulators and share feedback on visual quality, usefulness, and areas for improvement. We’ll use that feedback to expand the workflow, improve the renderer, and develop clearer guidance for future scenes.

The goal is not to make a scene look different for its own sake. The goal is to make real-world captures more useful as configurable environments – places that can support controlled variation without losing the visual fidelity of the original capture.

Real places as variation substrates

Relighting is an early step toward a larger idea: using the real world as a domain randomization substrate.

A single capture should not have to represent a single condition. The same place can become a training environment for different times of day, lighting conditions, and weather states.

That means more variation without more collection. More realism without giving up control.

We’re exploring multiple technical approaches to find the best combination of visual quality, robustness, and practicality for real-world captures. Our current implementation takes a direct path through Niantic Spatial’s real-to-sim pipeline, while ongoing experimentation helps us understand which methods generalize best across scenes and use cases.

The next phase depends on testing relighting in the environments where teams actually work. We’re inviting a small number of design partners with high-value scene variation use cases and representative environments to evaluate the workflow and help shape what comes next.