Virtual reality is all about having an immersive experience in a virtual world. One way to achieve that is to have experiences in your virtual world that users can relate to in the real world.

In this blog post, you will learn how to create an experience of taking regular pictures and selfies in VR. If you hand a phone and a selfie stick to a person, it is for certain that he/she will place the phone on the selfie stick and then tries to take pictures. This is true in Virtual Reality as well! You can create the best experience in VR by making sure that the selfie stick, the phone, and their functions work just as they do in real life.

This is one of the simplest mechanics which can be implemented, it requires a few of the components from the XR Interaction toolkit and four simple scripts. But before we begin let's look at the prerequisites.

1. Prerequisites

You must have a basic knowledge of installing the XR Interaction Toolkit, its components and properties, working with prefabs, and the basics of C# as well.

You should know how to set up a scene with a ground plane and an XR Rig. If not, this tutorial will help you get started with the XR Interaction Toolkit. Furthermore, you can learn about prefabs in this YouTube video.

Note: This was built and tested in Unity version 2021.1.14 and in the XR Interaction Toolkit version 1.0.0-pre.8.

While developing this project I made use of the XR Simulator for testing. I felt it speeds up the development process because putting the headset on and off, trying to get your controllers, and being very cautious with your surrounding is time-consuming. Also, for a person like me who wears glasses that process can be inconvenient. But in the end, once all the features are incorporated, it should be tested using the headset.

2. Setting up the GameObjects

In this section, we'll set up the model by adding a few components to it from the XR Interaction Toolkit. We'll also learn how to use a render texture to display the camera's output.

By the end of this section, we will have a selfie stick that can be grabbed and a "realistic" phone which can be attached to the selfie stick.

2.1 Importing the model to the scene

Let's start by importing the prefabs for the selfie stick. You can download the model from here.*  Or you can use any asset of your choice.

*( CC: "Selfie Stick" (https://skfb.ly/6WT6V) by Mason is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/). )

Once you have imported the models, follow the steps below:

• Drag and drop the prefabs into the hierarchy window.

• The imported prefab has two models, one is the selfie stick and the other the phone. Separate them by prefab unpacking and rename the prefabs as "SelfieStick "and "Phone".

2.2 Adding Grab Functionality

Note: The transform value changes depending on the scale/global position, the transform values seen in any of the below images/gifs might not be the same for you, you need to adjust the transform as per your scene.

Add the XR Grab Interactable component to both GameObjects, the SelfieStick, and the Phone. The RigidBody component gets added as well since it's a required component for the XR Grab Interactable.

• For this tutorial let's set up the physics of the rigid body as IsKinematic and turn off the Use Gravity option.
• The XRGrabInteractable component is not enough to make an object interactable. There is another component that's required and that's a collider. So, select the GameObject that has the handle and rename it as "Handle ". Add a Box Collider and adjust it accordingly.

• Then, add a Box Collider to the phone and adjust it accordingly as well.

2.3 Adding Socket Interaction

When the phone is held close to the holder of the selfie stick, the phone should get attached to the selfie stick. For us to do that we need to make use of the XRSocketInteractor component.

• In the hierarchy, select the GameObject that has a phone holder (The object that holds the camera onto the selfie stick) and rename it to "Holder". Create an empty GameObject as a child of the "Holder" and name it "Socket".
• Add the XRSocketInteractor component to it, then add a Box Collider and adjust the size accordingly. Also, check the IsTrigger box.

• How do we make sure that only the phone gets attached to the selfie stick and nothing else? Well, it's by making use of the layer mask! To do that, create a new layer called "Phone" and assign this layer to the GameObject Phone.

• Next, select the GameObject Socket, and in the XRSocketInteractor component, select the Interaction Layer Mask as Phone from the drop-down. In this way, only GameObjects are on the phone layer snap into the socket, and the rest of them are ignored.

• Also, drag and drop the GameObject Socket in the Attach Transform field. This will be useful to adjust the orientation of the phone when it gets snapped into the Holder.

2.4 Creating a realistic Phone

The GameObject Phone at its current state is just a 3D model. To make it work just like in real life, we need to add three components i.e two cameras and a quad. The two cameras act as the front and back camera while the quad forms the display for the camera's output.

Note: This can also be done in 3D modeling software like Blender. If you are able to, then feel free to do the same in 3D software and then import it.

Let's implement this by creating two new GameObjects as a child of the Phone and naming them as  "BackCamera" and "FrontCamera".

• Add the Camera component to the GameObjects BackCamera and FrontCamera. Adjust their transform accordingly.

• Create a render texture by right-clicking in the project window, Create → Render Texture. Rename it as "PhoneDisplay" and adjust its size as per your desired output size when the picture gets saved.

Note: The larger the size, the larger the amount of processing required. This can cause momentary lag, so you need to find the optimum size.

• Select the GameObjects FrontCamera and BackCamera. Drag and drop the Render Texture PhoneDisplay into the Output Texture of the Camera component.

• Create a Quad by right-clicking onGameObject Phone → 3D Objects → Quad. Rename it to Display.

• Create a material by right-clicking in the project window, Create→ Material. Rename it to Display. Change the Shader type to Texture (Unlit/Texture) by clicking on the drop-down and searching for texture.

• Drag and drop the Render Texture PhoneDisplay into the material.

• Drag and drop the Material Display into the GameObject Display. Resize and adjust the position to match the Phone's display. This will now display the output of either the Front Camera or Back Camera.

‍

<div class=callout><div class="callout-emoji">💡</div><p style="margin-bottom:0px;">Note: There is also another way of doing this. You can drag and drop the render texture on the GameObject Display and Unity will automatically create a Material for you.<p></div>

‍

2.5 Updating grab orientation

To make sure the phone and selfie stick is in the right orientation every time it's grabbed, we can make use of the Attach Transform feature of the XRGrabInteractable component.

• Create a new GameObject and rename it as AttachTransform. Drag and drop this GameObject into the XRGrabInteractable component of the phone.

• Click on the Play button to enter the game preview. Adjust the transform component of the GameObject AttachTransform such it's having the right orientation when grabbed.
• Copy the transform values and exit the game preview by clicking the Play button once again. Then, paste the values back into the transform component.

• Follow the same steps of adding AttachTransform to the GameObject SelfieStick.

• Finally, adjust the transform of the GameObject Socket such that the Phone snaps into the socket with the correct orientation.

3. Scripting additional features

Just like in the real world, the VR Phone should have the ability to click and save pictures. For that, we need to write a few scripts.

3.1 Functionality to save a picture

Let us start by creating the functionality that will save the picture in your local drive. For that create a new C# script, name it as SavePicture.

The following script takes the given camera and saves the output that is seen on the render texture.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class SavePicture : MonoBehaviour
{
    [SerializeField] private Camera cam;
    private int resWidth;
    private int resHeight;
    private bool takePicture = false;
   
    private void Awake()
    {
        resWidth = cam.targetTexture.width;
        resHeight = cam.targetTexture.height;
    }

    private void LateUpdate()
    {
        if(takePicture && cam.isActiveAndEnabled)
        {
            Texture2D snapShot = new Texture2D(resWidth, resHeight, TextureFormat.RGB24, false); //creating a texture
            cam.Render();
            RenderTexture.active = cam.targetTexture;
            snapShot.ReadPixels(new Rect(0, 0, resWidth, resHeight), 0, 0);
            byte[] bytes = snapShot.EncodeToPNG();
            string filename = SnapShotName();
            System.IO.File.WriteAllBytes(filename, bytes);
            Debug.Log("Snapshot taken");
            takePicture = false;
        }
    }
    
    private string SnapShotName()
    {
        return string.Format("{0}/Snapshots/snap_{1}.png", Application.dataPath, System.DateTime.Now.ToString("yyy-mm-dd_HH-mm-ss"));
    }

    public void TakePicture()
    {
        takePicture = true;
    }
}

• The script SavePicture has a function TakePicture() that will be called in the ClickPicture script.
• In the Project window create a new folder and name it Snapshots.

3.2 Functionality to click a picture

Now let's create the functionality that will detect the controller's trigger press to click the picture. To do so, create a new C# script and name it ClickPicture.

The following script takes the input from the controller and calls the function from the SavePicture script to save the picture on the local drive.

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.Interaction.Toolkit;

[RequireComponent(typeof(XRGrabInteractable))]
[RequireComponent(typeof(SavePicture))]
public class ClickPicture : MonoBehaviour
{
    private XRGrabInteractable interactable;
    private SavePicture savePicture;

    private bool isTriggerDown; //to check if trigger is pressed 
    private float triggerDownTime; //to capture the time duriation the user presses the trigger 
    private const float triggerDownTimeThreshold = 0.1f; // minimum duration before the photo is clicked [This can later be upgraded to have timer]

    private void Start()
    {
        interactable = GetComponent();
        savePicture = GetComponent();
        interactable.activated.AddListener(TriggerPulled);
        interactable.deactivated.AddListener(TriggerReleased);
        interactable.selectExited.AddListener(PhoneDropped);
    }

    private void Update()
    {
        if(isTriggerDown)
        {
            triggerDownTime += Time.deltaTime;
            if( triggerDownTime > triggerDownTimeThreshold)
            {
                savePicture.TakePicture(); // save a picture
                Debug.Log("took picture");
            }
        }
    }

    private void TriggerPulled(ActivateEventArgs arg0)
    {
        isTriggerDown = true;
    }

    private void TriggerReleased(DeactivateEventArgs arg0)
    {
        isTriggerDown = false;
        triggerDownTime = 0f;
    }

    private void PhoneDropped(SelectExitEventArgs arg0)
    {
        isTriggerDown = false;
        triggerDownTime = 0f;
    }

}

4. Joining the pieces

Alright, we have the SelfieStick, the Phone, and the additional scripts ready. Now it's time to stitch them all together.

To simplify the mechanics, we will have the back camera turned on and the front camera turned off by default. When the Phone is placed onto the holder of the selfie-stick, the front camera is turned on and the back camera is turned off, vice versa when removed from the selfie-stick.

• Select the GameObject FrontCamera and deactivate the component by unchecking the box.

• Use the Select Entered and Select Exited events of the Socket interaction to turn the camera on and off. This will also ensure that only one camera is active at a given time.

• Next, add the component ClickPicture to the GameObject Phone. The component SavePicture gets added automatically. Then, drag and drop the GameObject BackCamera into the field Cam of the Save Picture **component.

• Similarly, add the component ClickPicture to the GameObject SelfieStick. Then, drag and drop the GameObject FrontCamera into the field Cam of the Save Picture component.

With this, we have finished setting up our phone and selfie-stick with all the functionalities in VR. You go ahead and test it now!

5. Polishing

5.1 Mirrored Display

You might observe that while using the front camera, it renders the mirror image on the display. It's not possible to flip the camera, so we can correct this by flipping the display instead.

• To do so create a new C# script, name it SelfieDisplay. The following code will flip the display every time the function "FlipDisplay()" is called.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class SelfieDisplay : MonoBehaviour
{
    [SerializeField] private GameObject display;

    public void FlipDisplay()
    {
        display.gameObject.transform.localScale = new Vector3(display.gameObject.transform.localScale.x * -1, display.gameObject.transform.localScale.y, display.gameObject.transform.localScale.z);
    }
}

• Add the SelfieDisplay component to the GameObject Phone. Then, drag and drop the GameObject Display into the display field.

• Use the Select Entered and Select Exited events of the Socket interaction once again to flip the display.

Now the display will render the mirrored image for the front camera.

5.2 Fixing jitters

There is another possible issue you could notice! When the selfie stick is moved the phone passes through the holder. To fix this the GameObject Phone has to be made a child of the GameObject SelfieStick. It's also important to unparent the GameObject Phone when it's removed from the holder.

• Let's implement this by creating a C# script and name it PhoneHolder. The following code will use the SelectEntered and SelectExited events of the XRSocketInteractor to parent and unparent the GameObject Phone.

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.Interaction.Toolkit;

public class PhoneHolder : MonoBehaviour
{
    private XRSocketInteractor xRSocketInteractor;
    private GameObject phone;

    private void Start()
    {
        xRSocketInteractor = GetComponentInChildren();
        xRSocketInteractor.selectEntered.AddListener(PhoneHeld);
        xRSocketInteractor.selectExited.AddListener(PhoneReleased);
    }

    private void PhoneReleased(SelectExitEventArgs arg0)
    {
        phone = arg0.interactable.gameObject;
        phone.transform.SetParent(null);
    }

    private void PhoneHeld(SelectEnterEventArgs arg0)
    {
        phone = arg0.interactable.gameObject;
        phone.transform.SetParent(gameObject.transform);
        phone.transform.position = xRSocketInteractor.attachTransform.position;
    }

		private void OnDisable()
    {
        xRSocketInteractor.selectEntered.RemoveListener(PhoneHeld);
        xRSocketInteractor.selectEntered.RemoveListener((PhoneReleased);
    }

}

• Add the PhoneHolder component to the GameObject Holder.

Congrats! We have successfully fixed the jitters.

Conclusion

This tutorial not only taught you how to take pictures in VR and save them, but also taught you how you can use RenderTexture to display the camera's output. So what can you do next?.

You can either extend this project by adding or modifying some elements. For example, you can add a timer to take pictures or modify the size, orientation of the phone/selfie stick. You can also create a whole new project with a slightly different mechanism. For example, you could probably create a tablet with UI buttons. One to take pictures and another to switch between the two cameras.

There are many other things you can do with the render texture and a camera as well, like creating a mirror to see your avatar or casting your phone on a big screen, etc.

One main concern with VR features is to make them as smooth as possible. Everything that doesn't work perfectly or has hiccups, breaks the immersion. But keep in mind that in VR things can be completely different than in real life.

‍