Augmented Reality
AR/VR Streaming with ISAR SDK
by
immersive insiders
April 2022
8 MIN READ
Augmented Reality

AR/VR Streaming with ISAR SDK

by
immersive insiders
April 2022
8 MIN READ
FREE
Crucial Mistakes New
XR Devs Must Avoid
Level up your XR coding game
by dodging common mistakes!
Grab yourS NOW

So, what do we do if we want to run such a scene with high polygon and real-time lighting?

Luckily, there is Holo-Light, a  German company providing a remote rendering solution to that problem. They provide the ISAR SDK that allows us to either use our machine to run the scene and stream it to the device or run the scene on their cloud server and then stream that onto our device. In this blog, we’ll see how to stream the scene from our machine. Streaming the scene from their server will be covered in a future blog.

1. Prerequisites

There are only two things you need to make sure of:

  • The minimum version of Unity has to be 2019.4.x (tested with 2020.3.32f1)
  • The firewall on your machine has to be disabled or you can open the firewall for the TCP port 9999.
Note: While creating a new Unity project use the 3D render pipeline as there are some known issues with other render pipelines.

2. Setting Up Project For AR

In this section, we’ll learn how to set up our project for streaming into AR devices. As of now, the SDK supports HoloLens 2 and Android devices for AR. Although the current version is stable for HoloLens 2, it’s still under the BETA version for Android. So, in this blog, we’ll be focusing on the setup for HoloLens 2.

Note: Make sure you install the toolkit before installing ISAR SDK packages.

2.1 Downloading and Installing Toolkit

Let’s begin by downloading and installing MRTK in our project as it supports HoloLens 2.

  • To download MRTK, visit the GitHub page and scroll to the bottom and click on Microsoft.MixedReality.Toolkit.Unity.Foundation.2.7.3.unitypackage→ Once downloaded move it to a location of your choice.
MRTK
  • To import, open your Unity project → just drag and drop the downloaded file into the Project window → click on Import.
Import MRTK

2.2 Downloading and Installing ISAR SDK

Downloading and installing the ISAR SDK is simple. You can either download it from Unity’s Asset Store or you can download it from their Git repository.  As the Git repository has the latest version, we’ll use that for now

  • To download the SDK, visit the GitHub page and click on Code → Download ZIP. Once downloaded, unzip the folder and place it at a location of your choice.
ISAR SDK
  • Before installing the package we’ll have to remove the Version Control package to avoid any conflicts that could happen while importing the ISAR SDK.
    So, in your Unity project navigate to Windows → Package Manager → search for Version Control and click on Remove.
Version Control
  • To import the SDK into Unity:
  • In the Package Manager window click on the plus symbol ➕ → select Add packaged from disk..
  • In the ISAR-SDK folder, navigate to Packages → com.hololight.isar → select the package JSON file and click on Open.
Importing ISAR SDK
  • If you get a warning message on your Console window, you can ignore it.
  • Once again click on the plus symbol ➕ on the Package Manager window → select Add packaged from disk...
  • In the ISAR-SDK folder, navigate to Packages → com.hololight.isar.mrtk select the package JSON file **and click on Open.
Importing ISAR SDK for MRTK

With that, we have successfully downloaded and installed the ISAR SDK.

2.3 Setting up the Unity Project

For the SDK to work as intended we’ll have to set up a few things.

  • Go to File → Build Settings → Player Settings → XR Plug-in Management → check the box ✅ for ISAR XR.
XR Plug-in Management
  • In the Project window, navigate to Packages → ISAR Core → Runtime → Resources → open the remoting-config file.
remoting-config
  • Based on the documentation, different clients (HoloLens 2, Quest 2, Android) have different configurations. The width and the height represent the resolution of the device, the numViews represent the number of views and bandwidth represents the internet speed (which can be varied based on our internet speed).
    Since we are building for HoloLens 2, change/ensure the configurations are as follows:
SDK Configurations

2.4 Setting Up Scene

Now let’s set up our scene with MRTK and add a high poly model that we want to stream.

  • Configuring ISAR SDK with MRTK is simple, select ISAR → Configure MRTK. This will add the necessary GameObjects to the scene and configure it as well.
Configuring ISAR SDK with MRTK
  • To confirm that the scene has been configured correctly, select the MixedRealityToolkit GameObejct and make sure that:
  • Under the Camera tab, the Camera Settings Providers parameter contains XR SDK Camera Settings.
XR SDK Camera Settings
  • Under the Input tab, the Input Data Providers parameter contains ISAR XRSDK Device Manager and ISAR XRSDK Touch Device Manager
Input Data Providers
  • Now, let’s add a high polygon model to our scene. You can download and use any model of your choice or use the same one as I did from here. The model I am using has 508.7k polygons.
  • Import the asset into Unity by dragging and dropping it into the Project window. Then, you can move the prefab into your scene.
  • We’ll duplicate the prefab three times so that we have more polygons being rendered at a time → position them one beside the other → create an empty GameObject and name it as Models → move all the four prefabs into this GameObejct.
Creating scene
  • Note: Sometimes the model may come with a camera component, in such cases, make sure to disable the camera component or remove it as well.
Imported models
  • Finally, we’ll add a script that will rotate the models. Create a new C# script, name it Rotate and copy the following code:

using UnityEngine;

public class Rotate: MonoBehaviour
{
    [SerializeField] private float speed = 40;

    void Update()
    {
        transform.Rotate(Vector3.up, Time.deltaTime * speed);
    }
}


  • Add the script to the Models GameObject and hit play to test it.
Testing

With that we have set up our scene, next we’ll see how we can configure the Client on HoloLens 2

2.5 Installing ISAR Client

There are two ways by which we can install the ISAR Client on HoloLens 2. We can either install the app package contained in the repository or install it from the Microsoft store.

  • You can find the app package by opening your file explorer and navigating to ISAR-SDK-Trial-main\Clients\ISARClient_HoloLens2. If you are not sure as to how to install the package from pc then you can check out this video.
ISAR Client
  • The easier way would be to download and install the client from the Microsoft store. All you need to do is select Get and select your HoloLens2 device
UISliders

Now that we have the Client installed we can proceed and see how to stream the assets into HoloLens 2.

2.6 Streaming The Scene

To stream the scene from our machine to the client, we need to first disable the firewall. Then we can play the Unity scene and stream it into the client.

  • To disable the firewall on your machine:
  • Search for Windows Security Settings and open it.
UISliders
  • Click on Firewall & network protection → Public network → click the Microsoft Defender Firewall toggle to turn it off
UISliders
  • You will get a popup asking if you want to allow the app to make changes, click on Yes.
UISliders
  • Once we have the firewall disabled you can hit the play button ▶️ on the Unity Editor and open the ISAR client application on your HoloLens 2.
  • On the ISAR client app, key in your local IP address and click on Connect. If you are not sure how to find your IP address you can check this out.

With that, we were successfully able to stream the scene with high poly count seamlessly onto HoloLens2.

2.7 Comparison

Generally, HoloLens 2 can handle up to 100,000 polygons and our scene has more than 2000,000 polygons. Just for comparison, you can build and run the scene locally on the HoloLens 2 and check the FPS. Later check the FPS when it’s streamed from the machine. You will definitely see a difference. Here is our comparison: the local build runs at 8 or 9 FPS and when streamed using ISAR SDK it runs at 45-47 FPS.

With that, we have successfully learnt how to stream experiences into AR devices. Up next, we’ll see how to stream it into VR devices

3. Setting Up Project For VR

In this section, we’ll see how to set up the project for VR. As of now, the SDK is supported on Oculus Quest 2. The setting up part for VR is the same as setting up for AR, there is just one difference! And that is the values entered in the remoting-config file.

  • In the Project window, navigate to Packages → ISAR Core → Runtime → Resources → open the remoting-config file.
  • Based on the documentation, change/ensure the configurations are as follows for Quest 2:
Configuration for Quest2
  • Also, we’ll have to install the client on our device either through Oculus App Lab or by sideloading the APK file. You can find the app package by opening your file explorer and navigating to ISAR-SDK-Trial-main\Clients\ISARClient_OculusQuest2. If you are not sure how to sideload APKs into Quest 2 then check this out.

You can now hit the play button → launch the client on Quest 2 → enter your local IP address and view the streamed assets.

4. Updating the ISAR SDK

In future when the SKD gets updated we’ll have to first remove the existing SDK and then reimport the latest one.

To remove previous versions of ISAR SDK from the project

  • Open Package Manager in the Unity editor by clicking on Window → Package Manager.
  • Search for the ISAR Core package and click on Remove
  • Similarly, search for the ISAR MRTK Extensions package and click on Remove.

To add import the latest SDK follow the same steps mentioned in section 2.2.

5. Conclusion

We have learnt to use the ISAR SDK along with MRTK to stream the assets into HoloLens 2 and Quest 2. The SDK comes in real handy when we want to see models/experiences in VR without losing any visual quality. In the next blog, we’ll see how we can use the XRTK along with ISAR SDK to stream a VR experience into Quest 2.Have you ever tried to run a scene with high polygon models and real-time lighting on your AR/ VR devices? I have tried it and I had the worst experience ever. If you haven't tried it yet, then you should know that running such a scene on a standalone device will cause the FPS to drop. You will see lag, probably get motion sick, it can also cause the device to overheat, etc.

Thanks for reading this blog post 🧡

If you've enjoyed the insights shared here, why not spread the word? Share the post with your friends and colleagues who might also find it valuable.

Your support means the world to us and helps us create more content you'll love.

Read more by this author

Continue reading