How to Create Datasets for Playback

With NSDK's Recording feature, you can record the real-world location of your AR application for playback and testing in the Unity editor. You can create a playback dataset by using the Recording sample project or by recording through the API.

Recording data

Recording formats

NSDK can produce two versions of a scan recording:

1. Raw Scan format: By default, AR Scanning Manager will write frames directly to disk.

• This is the format that Scan Reconstruction accepts.
• The files are saved to the directory at ScanStore.SavedScan.ScanPath. Metadata is stored in .pb files.

2. Playback format: Optionally, the recorded frames can be exported to a compressed .tgz archive.

• This is the format that Playback and VPS accept.
• The sequence is archived to ScanStore.SavedScan.ScanBasePath in a .tgz. Metadata is stored in a capture.json file.

This walkthrough demonstrates how to produce the second format for use with Playback.

File locations

By default, scan recordings will be saved to these paths. Setting AR Scanning Manager's Scan Path field to a custom location will override this. In Unity apps, the saved location is available in the ScanStore.SavedScan.ScanBasePath property[ScanStore.SavedScan.ScanBasePath.

Platform	Unity SDK	Native SDK
iOS	{App Files}/scankit/	{App Files}/scankit/
Android	/sdcard/Android/data/{app.package.name}/files/scankit/	/sdcard/Android/data/{app.package.name}/files/scankit/
macOS	/Users/{Username}/Library/Application Support/{CompanyName}/{ApplicationName}/scankit/	{Application path}/data/scankit/
Windows	C:\Users\{Username}\AppData\LocalLow\{CompanyName}\{ApplicationName}\scankit\	C:\Users\{Username}\AppData\Local\ardk_data\scankit\

To retrieve the files from an iOS app in Finder, add UIFileSharingEnabled and LSSupportsOpeningDocumentsInPlace to the app's info.plist. To retrieve the files from an Android app, use adb pull with the appropriate path.

Using the Scanning Sample to Record

Prerequisites

To scan using the Recording sample project, you will need to download it and build it to your device. For more information, see the Recording sample project.

Steps

To record a playback dataset using the sample:

1. Open the recording sample on your device. If running on iOS, connect to the internet. When you are ready to record the dataset, tap Start.
2. Wait for the recording to begin. When scanning with Unity on iOS, there is a brief delay the first time that scanning starts while the SDK downloads a model file to convert Unity's MSL altitude values to WGS84.
3. Record the environment using your device's camera. Keep in mind the following:

• Keep your device in Portrait Mode.
• Hold your device as steadily as possible to reduce errors in the dataset.
• The sample saves dataset recordings every minute. If you continue recording after one minute, the app will start a new recording. (For example, if you record for 3.5 minutes, the sample creates four separate datasets.)

3. Tap Stop to finish recording. At this point, you have a recording in Raw Scan format.
4. When you are ready, tap Export to convert your most recent Raw Scan recording to a Playback format dataset.
5. After the export is complete, the sample will display a path to the archived dataset. Remember this path and proceed to Accessing Your Recorded Data on the Device.

note

For Unity iOS builds, NSDK is configured to convert the MSL altitude provided by Unity to WGS84. This conversion requires a one-time download. The first time that scanning runs on iOS, the device must have an active internet connection and will experience a brief delay while the 20 MB model file downloads.

Using the API to Record Datasets

If the Recording sample is too limited for your project, you can create your own recording app instead.

Prerequisites

You will need a Unity project with NSDK installed and a set-up basic AR scene. For more information, see Setting Up the Niantic SDK for Unity and Setting up a Basic AR Scene.

Steps

1. Enable Scanning in NSDK Settings:
   1. In the NSDK top menu, select Settings.
   2. In the Inspector window, check the box labeled Scanning.

1. In your AR Scene, add an AR Scanning Manager to the scene, then disable it:
   1. Select the ARSession GameObject.
   2. In the Inspector window, click Add Component, then add an AR Scanning Manager to it.
   3. Un-check the box next to AR Scanning Manager (Script) to disable it.

caution

Enabling Record Estimated Depth to record NSDK depth buffers on non-lidar devices will limit the AR Scanning Manager's recording FPS to the update rate of the depth feature, regardless of AR Scanning Manager's Recording Framerate selection.

To change the update rate of the depth feature from its default rate of 10 FPS, add an Nsdk Occlusion Extension component on the same Game Object as an AR Occlusion Manager and set the Target Frame Rate.

1. Create two buttons in your scene; one to start the recording, the other to stop it.
   1. Right-click in the Hierarchy, then, in the UI menu, select Button. Rename the Button "Record".
   2. Repeat the previous step, naming the new button "Stop".
   3. For each button, expand its GameObject and select the Text sub-object, then change the Text field to "Record" or "Stop", as appropriate.
   4. Move the buttons in the scene view to where you would like them.
2. Create scripts to drive the recording, then connect them to the buttons:
   1. In the Project window, right-click in the Assets folder, then select C# Script from the Create menu. Name the new script StartScript. Create another script and name it RecorderInput. Populate the scripts with the following code:

Click to reveal StartScript.cs

  using System.Collections;
  using System.Collections.Generic;
  using UnityEngine;

  public class StartScript : MonoBehaviour
  {
      public void Start() {
          // get permission to use location data if on Android
          // then enable location and compass services
  #if UNITY_ANDROID
          if (!Permission.HasUserAuthorizedPermission(Permission.FineLocation))
          {
              var androidPermissionCallbacks = new PermissionCallbacks();
              androidPermissionCallbacks.PermissionGranted += permissionName =>
              {
                  if (permissionName == "android.permission.ACCESS_FINE_LOCATION")
                  {
                      Start();
                  }
              };

              Permission.RequestUserPermission(Permission.FineLocation, androidPermissionCallbacks);
              return;
          }
  #endif
          Input.compass.enabled = true;
          Input.location.Start();
      }
  }

Click to reveal RecorderInput.cs

using System.Collections;
using System.Collections.Generic;
using NianticSpatial.NSDK.AR.Scanning;
using NianticSpatial.NSDK.AR.Scanning;
using UnityEngine;

public class RecorderInput : MonoBehaviour
{
    [SerializeField] private ARScanningManager _arScanningManager;

    public async void StopRecordingAndExport() {

        // save the recording with SaveScan()
        // use ScanStore() to get a reference to it, then ScanArchiveBuilder() to export it
        // output the path to the playback recording as a debug message
        string scanId = _arScanningManager.GetCurrentScanId();
        await _arScanningManager.SaveScan();
        var savedScan = _arScanningManager.GetScanStore().GetSavedScans().Find(scan => scan.ScanId == scanId);
        ScanArchiveBuilder builder = new ScanArchiveBuilder(savedScan, new UploadUserInfo());
        while (builder.HasMoreChunks())
        {
            var task = builder.CreateTaskToGetNextChunk();
            task.Start();
            await task;
            Debug.Log(task.Result);   // <- this is the path to the playback recording.
        }
        _arScanningManager.enabled = false;
    }

    public void StartRecording() {
        _arScanningManager.enabled = true;
    }

}

1. In the Hierarchy, right-click, then select Create Empty. Name the new GameObject "RecordScripts". With RecordScripts selected, in the Inspector window, click Add Component and add StartScript. Also add RecorderInput, making sure to pass the AR Scanning Manager from the Hierarchy.

1. Select the Record button in the Hierarchy, then add a script to the OnClick field in the Inspector by dragging the RecordScripts GameObject into the field and selecting RecorderInput.StartRecording. Do the same for the Stop button, selecting RecorderInput.StopRecordingAndExport at the end.

1. Build to your device and test it out!

Accessing Recorded Data on the Device

Follow the instructions for your device's operating system:

Exporting from iOS

1. Enable developer mode on your device:
   1. Open the Settings app, then open the Privacy and Security menu and enable Developer Mode.
2. Connect your device to Xcode.
3. Open the Window top menu, then select Device and Simulators and find the sample app.
4. With the app highlighted, click the three dot menu and click Download Container.

1. Once the container downloads to your machine, right-click on the package, then click Show Package Contents.
2. Navigate to the path from the recording output (for example: AppData/Documents/scankit/(ID of your scan)/chunk_0.tgz).
3. Copy the archive to your machine, then unzip it.

Exporting from Android

1. Connect your device to your development machine.
2. Get the files from your device:
   1. MacOS: Open Android File Transfer, then navigate to the path from the recording output. You may also choose to use Android Studio's Device Explorer, or another tool of your preference.
   2. Windows: When the dialog pops up asking what you want to do, select File Transfer, then navigate to the path from the recording.
3. Copy the archive to your machine, then unzip it.

Using the Playback Data in Unity

To add a playback dataset to Unity:

1. Open Project Settings from the Edit menu, then scroll down to XR Plugin Management and select Niantic SDK.
2. Enable Editor Playback, then input the absolute path to your dataset in the Dataset Path field.
3. If using on-device playback, make sure the dataset is copied into the Assets/StreamingAssets directory of your Unity project before building.