How to Create Spatial Maps and Place AR Content with Immersal SDK

If you've ever wondered how to integrate AR content into your environment. By the end of this blog, you'll be equipped to:

• Create Spatial Maps: Capture and map your environment.
• Develop AR Experiences: Implement features like AR multi-user interactions, indoor navigation, and overlaying BIM 
  models to visualize hidden structures.
• Explore WebXR: Extend your AR projects to web-based platforms.

Immersal

Immersal, part of Hexagon, focuses on augmented reality (AR) technology, offering solutions like the City-Scale Visual Positioning System to blend digital and physical worlds. This enhances Hexagon's digital reality offerings by enabling interactive, location-based experiences.

What’s next?

It might sound like a lot, but don't worry, it's easier than you think. You'll be thrilled to know it only takes three simple steps:

Device Compatibility

For this blog we build it for Android phones, but you can build the app using IOS and other devices including AR headsets (MagicLeap, HoloLens or XREAL) with just a few changes to the project settings.

Step 1: Map Your Room

The first step is all about mapping your room. You'll need to download the Immersive Mapper app from either the Google Play Store here or the App Store here. As it downloads, head over to the developer website here to create an account. We'll use these credentials to log in to the app and also while developing in Unity.

Once the app is downloaded, open → press the button center-down → You will a button M/R

Immersal, offers two modes in its AR technology: Manual Mode (M) and Realtime Mode (R). Manual Mode allows users to create and position digital content within a physical space by manually mapping the environment. Realtime Mode enables live, dynamic mapping and interaction, providing seamless integration of digital elements with the physical world, thus enhancing interactive and location-based AR experiences.

Which mode do you choose and when?

1. Manual Mode: You need precise control over the mapping process and have the time to set up the environment beforehand. It's ideal for static environments where accuracy is crucial, such as museums or exhibitions, where digital content needs to be consistently positioned in specific locations.
   1. however, It requires time-intensive setup and mapping, making it less suitable for large, dynamic environments, and lacks adaptability to changes once mapped.
      
2. Realtime Mode: You require dynamic, on-the-fly mapping and interaction, such as in outdoor events or rapidly changing environments. This mode is suited for applications where quick setup and adaptability are essential, like interactive tours or games, allowing digital content to align with the physical space in real-time.
   1. It may sacrifice precision for flexibility, being more resource-intensive and less accurate in complex environments compared to manual mapping.

Evaluating Your Map's Quality

Once you've captured your environment using the Immersal Mapper app, it's time to evaluate the quality of your map. Here's how you can do it:

Using Manual Mode

• Capture and Construct: For this blog, we used manual mode to capture the studio room. After the map construction is complete, download the map and localize it in AR to check its quality.
• Quality Check: If the map localizes within a few attempts, you have a good quality map.
  

Using Real-Time Mode

• Upload and View: If you used real-time mode, upload the map to your portal. Click on "View Your Maps" to see a list of all captured maps.
• Evaluate Map Quality:
  • Sparse Poly: If you can only view the sparse poly, the map quality is not optimal. It lacks enough feature points to generate a dense poly or a GLB.
  • Dense Poly and GLB: If you can view both the sparse and dense polys but encounter a "mesh fail" when trying to view the GLB, the map quality is insufficient for generating a GLB file.

Step 2: Setting Up and Writing some Code

To start developing your AR application, we'll need to set up a new Unity project. Follow these steps to get started:

1. Create a New Unity Project

• Unity Version: Ensure you are using Unity Editor version 22.3 LTS or above.
• Project Setup:
•  
  • Select the 3D Core render pipeline.
  • Give your project a name.
  • Click Create to start the project.
    

2. Add the Immersal SDK

• Open Package Manager: Navigate to Window → Package Manager.
• Add Package:
  • Click the plus symbol (+).
  • Select Add package from git URL.
  • Paste the URL provided in the description below and click Add.
• Enable Input System: During installation, you might see a warning about the new input system not being enabled. Click Yes to enable it.

3. Configure Project Settings

• Project Validation: After installing the SDK, you'll see the Immersal project validation window. It will show recommended settings specific to your target platform.

• Build Settings:

  • Navigate to File → Build Settings.
  • Select Android as your build platform. If you're building for iOS, select iOS and switch the platform.
  • Click Fix All to apply the recommended settings.

• Set ARCore as XR Plugin Provider:

  • Click on Fix.
  • Select Google ARCore as the plugin provider.

4. Set Up Your Scene

Now that your Unity project is set up with the Immersal SDK, it's time to configure your scene for AR development. Follow these steps to get started:

• Remove Main Camera: Select the main camera in the hierarchy and delete it. This step is crucial as we'll be adding AR-specific components that handle camera functionality.
• Add AR Session and XR Origin:
  • Right-click in the hierarchy, navigate to XR, and add AR Session and XR Origin.
  • The AR Session component manages the lifecycle of your AR experience, ensuring it runs smoothly on your device.
  • The XR Origin component enables device tracking and transforms trackables into Unity's coordinate system.
• Track Pose Driver:
  • Open the child object of XR Origin and select the main camera.
  • Scroll down to find the Track Pose Driver component.
  • This component is responsible for obtaining the position and rotation of your device, allowing the XR Origin to transform trackable objects and features to their final positions within the Unity scene.
• Understanding the Components
  • AR Session: Handles the initialization and management of AR features on your device.
  • XR Origin: Provides a reference point for AR tracking, ensuring that virtual objects are accurately placed in the real world.
  • Track Pose Driver: Captures the device's movement and orientation, crucial for maintaining the alignment of virtual content with the physical environment.

5. Integrating the Immersal SDK and Setting Up VPS Features

With your Unity project set up, it's time to integrate the Immersal SDK and configure the VPS (Visual Positioning System) features to place virtual content. Here's how you can do it:

5.1 Add the Immersal SDK Prefab

• Search for Immersal: In your project window, set the search filter to "All" and search for "Immersal"
• Add SDK Prefab: Select the Immersal SDK prefab and add it to your scene. This prefab includes AR Foundation support, which configures the device and retrieves necessary camera data.

5.2 Understand the Components

• Localization Methods: The prefab includes two localization methods that define how localization should occur.
• Localizer Component: Manages and configures the localization methods.
• Scene Updater Component: Adjusts the position from the localizer and updates the scene accordingly.
• Spatializer Component: Interprets data such as device status and localization results, constructing a tracking status to describe positioning quality.
• Immersal Session Component: Connects all components, managing a continuous loop to process data based on internal logic and configuration.

5.3 Set Up XR Space

• Create XR Space: Create an empty GameObject named "XR" and add the XR space component.

• Add Data Processors:

  • Create a child GameObject named "Pose Filter" and add the pose filter component.
  • Create another child GameObject named "Pose Smoother" and add the pose smoother component.
  • Add both components to the XR space's data processor list.

5.4 Download the Spatial Map

• Create XR Map: Create an empty GameObject as a child of XR space, name it XR Map and add the XR map component.
• Login to Immersal SDK: Use your Immersal credentials to log in within Unity. Upon successful login, you'll see a token.
• Map ID: Copy the map ID from the Immersal developer portal, paste it in Unity, check Map File Visualization and click Download

6. Configure and Test

• Configure Localization: Decide if you want the localization method to be device-localized or server-localized. Similarly, choose if the map data should be embedded or downloaded.
• Render Settings: Select the XR map's child object with the XR map visualizer, choose the render mode (Editor or Runtime), and adjust point color and size as needed.
• Build and Test: Save your scene, enable developer mode on your device, connect it to your computer, navigate to File → Build Settings, add the open scene, select your device from the dropdown, and click "Build and Run"

7. Adding AR Content

Now that our app can localize the spatial map, it's time to add virtual content to your AR scene. Follow these steps to enhance your AR experience:

7.1 Create a Content Holder

• Create Content Object: Right-click on the XR space in the hierarchy, create an empty GameObject, and name it "Content"

7.2 Import Core Samples

• Access Package Manager: Navigate to Window → Package Manager.
• Select Immersal SDK: Find the Immersal SDK in the list.
• Import Core Samples: Go to the Samples tab and import the core samples.

7.3 Add Virtual Content

• Locate AR Diamond: Inside the imported samples, navigate to Core Samples → Prefabs → Content Placement.
• Add AR Diamond: Drag and drop the AR Diamond prefab into the "Content" GameObject you created.
• Position the Diamond: Move and align the diamond with your spatial map. For example, place one diamond above your camera and duplicate it to place another near your laptop.

7.4 Adjust Point Cloud Visibility

• Hide Point Clouds: If you don't want to see the point clouds in your application, navigate to the XR map, select its child object, and set the render mode to "Editor Only."

7.5 Save and Build

• Save Your Scene: Ensure all changes are saved.
• Build the Application: Navigate to File → Build Settings, add the open scene, and build the application to your device.

8. Localization Status

• Create UI Canvas: Right-click in the hierarchy and create a new UI Canvas.
• Import Utilities: Navigate to Samples → Immersal SDK → [Version] → Core Samples → Prefabs → Utilities.
• Add UI Elements: Drag and drop the Localization Status Text and Pose Indicator into the Canvas.
• Import TextMesh Pro: When prompted, import the TextMesh Pro essentials.
• Display Information: The UI will now show the number of successful localizations versus attempts and display tracking quality in the top-left corner.

9. Persistent Spatial Anchors

Persistent spatial anchors allow virtual content to remain in the same position across app sessions. Here's how to implement them: