Ready Player Me Avatars in AR

Previously we learned to integrate Web View on the mobile app to create an avatar and add it to the scene. If you haven’t checked out the first part of this series, we highly recommend checking it out, since the skills, knowledge and project settings from it are required here as well.

In this blog post, we’ll learn how to augment the imported avatar on a horizontal plane. We’ll also add animations and move them around by tapping at the desired location on the plane.

1. Installing Packages and AR Setup

Let’s begin by downloading the AR packages.

• Click on Windows → Package Manager → select the ARFoundation v4.1.9 and click Install .
• Also, select ARCore XR Plugin v4.1.9 and click Install.

• Close the packager manager and now we can enable AR Core by navigating to Edit → Project Settings. Here, select the XR Plug-in Management tab and check ✅ AR Core.

2. Setting up the scene for AR:

Now, let’s set up the scene for AR by adding a few components.

• In the hierarchy, right-click and select XR → AR session. This will create a new AR Session GameObject with the AR Session component and the AR Input Manager component
• Next, right-click in the hierarchy window once again and select XR → AR Session Origin. This will create a new AR Session Origin GameObject with the AR Session Origin component. It also creates an AR Camera Gameobject as a child with the required components like a Camera, AR Pose Driver, AR Camera Manager, **and AR Background.
• Additionally, add the AR Plane Manager component to the AR Session Origin GameObject

• The AR Plane Manager component requires a plane prefab. This prefab will be the visual representation of the planes getting detected.
  So, to create a plane prefab:
• Right-click in the hierarchy window and select XR → AR Default Plane.
• Drag and drop it in the project folder to convert it as a prefab.
• From the project folder, drag and drop the AR Default Plane prefab into the AR Plane manager component.
• Finally, delete it from the scene.

• Later on, we'll be detecting horizontal planes and when a plane of a certain size is found, we’ll have to assign it a tag. So, create a tag called Floor by navigating to Edit → Project settings → Tags and Layers → under Tags, click on the plus ➕ button to add a new tag and name it as Floor.

3. Downloading Animations

The Ready Player Me SDK comes with an inbuilt idle animation and a walking animation. We need to download the animation to rotate the avatar either on the left side or the right side.

• Before we download the animation, create an empty folder and name it Animations in the Project window.
• On your web browser, visit the Mixamo website and upload your avatar → search for turn animation → select the appropriate animation for right turn→ click on Download → select the Format as FBX for Unity and save it on your device.
• Then, check the box ✅ for Mirror and download it once again → Rename it from Right turn to Left turn and save it on your device.

Note: If you are still not sure how the animation is downloaded, feel free to check out this video.

• Now switch back to Unity and import the downloaded FBX files in the Animations folder. You can do that either by dragging and dropping or simply right-click in the Project window, select Import New Asset and then select the two files.

• As we want the animation to match with the Avatar’s rig, select both the models → click on the Rig tab → select the Animation Type as Humanoid → select the Avatar Definition as Copy From Other Avatar → select MaleAnimationTarget V2 or FemaleAnimationTarget V2, as per your choice → click Apply.

• Select the models one at a time → click on the Animation tab → check the box ✅ for Loop Time

4. Setting Up The Animation Controller

We’ll have to create and set up an Animation controller with different boolean parameters. With the help of those parameters, we’ll be able to transition between different animations states.

• Navigate to Assets → Plugins → Wolf3d Ready... → Resources →AnimatorControllers.
• Create an Animator controller by right-clicking in the Project window → Create → Animator Controller → name it AvatarAnimator

• Open the AvatarAnimator→ click on the Parameters tab → add a Bool parameter by clicking on the plus sign + and name it isMoving. Similarly, add two more bool parameters and name them as isTurningLeft and isTurningRight.

• We need to create four animation states for our avatar and those are Idle state, Walking state, Turning Left state, and Turning Right state. To do that :
• Right-click on the base layer→ Create State → Empty→ name it as Idle → add the Breathing Idle animation in the motion field.
• Create another empty state → name it as Walking → add the Walking animation in the motion field.
• Similarly, create two more empty states → name it as LeftTurn and RightTurn → add the LeftTurn and RightTurn animation in the motion fields respectively.

• Now, to make transitions between the states:
• Right-click on the Idle state → select Make Transition and then click on the Walking state. Next, right-click on the Walking state, click on Make Transition and then click on the Idle state.
• Similarly, make the transitions between the Idle state and the LeftTurn state, and also between the Idle state and the RightTurn state.

• To add the parameter which will define the condition as to when the transition should happen :
• Click on the transition arrow from the Idle to the Walking state → in the Inspector window, uncheck Has Exit Time box → click on the Plus button ➕ under Condition and make sure the parameter chosen is isMoving and the value is true.
  Next, click on the transition arrow from the Walking to the Idle state → uncheck Has Exit Time box → click on the Plus button + and make sure the parameter chosen is isMoving and the value is false.
• Click on the transition arrow from the Idle to the LeftTurn state → in the Inspector window, uncheck Has Exit Time box → click on the Plus button ➕ under Condition and make sure the parameter chosen is isTurningLeft and the value is true.
  Next, click on the transition arrow from the LeftTurn to the Idle state → uncheck Has Exit Time box → click on the Plus button + and make sure the parameter chosen is isTurningLeft and the value is false.
• Click on the transition arrow from the Idle to the RightTurn state → in the Inspector window, uncheck Has Exit Time box → click on the Plus button ➕ under Condition and make sure the parameter chosen is isTurningRight and the value is true.
  Next, click on the transition arrow from RightTurn to Idle state → uncheck Has Exit Time box → click on the Plus button + and make sure the parameter chosen is isTurningRight and the value is false.

5. Scripting

To place the avatar on the detected surface and move them to the touch location, we’ll have to write a few scripts. We already have a script to import the avatar from the Ready Player Me website using web view. We need to modify it a little to meet our requirements.

5.1 Updating Avatar Importer

• Open Avatar Importer script and add the following code above the Start method:

public UnityEvent OnAvatarStored;

		public GameObject ImportedAvatar
    {
        get
        {
            return importedAvatar;
        }
        set
        {
            importedAvatar = value;
        }
    }

‍

• Modify the ImportAvatar and StoreAvatar methods as follows:

private void ImportAvatar(string url)
    {
        AvatarLoader avatarLoader = new AvatarLoader();
        avatarLoader.LoadAvatar(url, null,StoreAvatar);
    }

    private void StoreAvatar(GameObject avatar, AvatarMetaData meta)
    {
        importedAvatar = avatar;
        importedAvatar.transform.localScale = Vector3.one * 0.2f;//scaling it to 0.2 of its original size
        importedAvatar.SetActive(false);
		OnAvatarStored.Invoke();
    }

‍

• In the ImportAvatar method, the reason for writing the LoadAvatar API with a different overload is because earlier we used to receive the avatar GameObject when it was imported but we need the avatar when it's loaded.
• In the StoreAvatar method, we are shrinking the size to 2/10th its original size so that It can be easily seen In AR. You can scale it as per your choice.
• We’ll also be invoking an event when the avatar gets stored.
• The entire script should look like this:

using UnityEngine;
using UnityEngine.Events;
using Wolf3D.ReadyPlayerMe.AvatarSDK;

public class AvatarImporter : MonoBehaviour
{
    [SerializeField] private WebView webView;

    private GameObject importedAvatar;

	public UnityEvent OnAvatarStored;

    public GameObject ImportedAvatar
    {
        get
        {
            return importedAvatar;
        }
        set
        {
            importedAvatar = value;
        }
    }

    private void Start()
    {
        webView.CreateWebView();
        webView.OnAvatarCreated = ImportAvatar;
    }

    private void ImportAvatar(string url)
    {
        AvatarLoader avatarLoader = new AvatarLoader();
        avatarLoader.LoadAvatar(url, null,StoreAvatar);
    }

    private void StoreAvatar(GameObject avatar, AvatarMetaData meta)
    {
        importedAvatar = avatar;
        importedAvatar.transform.localScale = Vector3.one * 0.2f; //scaling it to 0.2 of its original size
        importedAvatar.SetActive(false);
		OnAvatarStored.Invoke();
    }
}

‍

5.2 AR Plane Detection

Create a new script named ARPlaneDetection and copy the following code. The code will make a list of all the planes ARPlaneManager has detected, filter out the 1st plane that has an area of more than 0.5 units i.e 0.5m square.

Once that plane is found,

• The tag “Floor” will be assigned to the found plane
• The AR Plane Manager component will be disabled so that further planes are not detected.
• The already detected planes will be disabled.
• Finally, this component will be disabled as well to unsubscribe from the events.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;

public class ARPlaneDetection : MonoBehaviour
{
    [SerializeField] private ARPlaneManager arPlaneManager;

    private bool isFloorPlaced;

    private List foundPlanes = new List();
    private ARPlane foundPlane;

    private void OnEnable()
    {
        arPlaneManager.planesChanged += PlanesChanged;
    }

    private void PlanesChanged(ARPlanesChangedEventArgs obj)
    {
        if (obj != null && obj.added.Count > 0)
        {
            foundPlanes.AddRange(obj.added);
        }

        foreach (ARPlane plane in foundPlanes)
        {
            if (plane.extents.x * plane.extents.y >= 0.5f && !isFloorPlaced)
            {
                isFloorPlaced = true;
                foundPlane = plane;
                foundPlane.tag = "Floor";
                DisablePlanes();
            }
        }
    }

    private void DisablePlanes()
    {
        arPlaneManager.enabled = false
        foreach (var plane in arPlaneManager.trackables)
        {
            if( plane != foundPlane)
                plane.gameObject.SetActive(false);
        }
        this.enabled = false;
    }

    private void OnDisable()
    {
        arPlaneManager.planesChanged -= PlanesChanged;
    }
}

As we are disabling the AR Plane Manager, the shape and size of the plane get fixed. The plane will not grow bigger even if you scan other areas. As AR plane scanning is not perfect, leaving the AR Plane Manager enabled will allow the found planes to grow but it will detect new planes which can overlap the existing one. When you tap at that particular location the avatar will not move, this can break the experience.

5.3 Tap To Place:

Create a new script named ARTapToPlace and copy the following code. The code will store the new touch position on the found plane and invoke an event. Later on, this new touch position will be used to move the avatar to that position using the event.

using UnityEngine;
using UnityEngine.Events;

public class ARTapToPlace : MonoBehaviour
{
    [SerializeField] private AvatarImporter avatarImporter;
    [SerializeField] Camera arCam;

    private Vector3 newPosition = Vector3.zero;
    public Vector3 TouchPosition { get => newPosition; }

    public UnityEvent OnNewTouch;

    private void Update()
    {
        if (Input.touchCount > 0)
        {
            Touch touch = Input.GetTouch(0);

            if (touch.phase == TouchPhase.Began)
            {
                Ray ray = arCam.ScreenPointToRay(touch.position);
                RaycastHit hitAnything;

                if (Physics.Raycast(ray, out hitAnything, Mathf.Infinity))
                {
                    StorePosition(hitAnything);
                }
            }

        }
    }

    private void StorePosition(RaycastHit hitAnything)
    {
        if (hitAnything.transform.gameObject.CompareTag("Floor"))
        {
            if (avatarImporter.ImportedAvatar.activeSelf == false)
            {
                avatarImporter.ImportedAvatar.SetActive(true);
                avatarImporter.ImportedAvatar.transform.rotation = hitAnything.transform.rotation;
                avatarImporter.ImportedAvatar.transform.position = hitAnything.point;
            }
            else
            {
                newPosition = hitAnything.point;
                OnNewTouch.Invoke();
            }
        }
    }
}

5.4 Avatar Animator Controller:

Create a new script named AvatarAnimationController and copy the following code. The code will be used to start and stop the appropriate animation when the respective methods are called. i.e start turn animation when the avatar has to turn, walk when the avatar has to move, etc.

using UnityEngine;

public class AvatarAnimationController : MonoBehaviour
{
    [SerializeField] private AvatarImporter avatarImporter;
    [SerializeField] private RuntimeAnimatorController avatarController;

    private Animator animator;

    private void Start()
    {
        avatarImporter.OnAvatarStored.AddListener(AssignAvatarController);
    }

    private void AssignAvatarController()
    {
        animator = avatarImporter.ImportedAvatar.transform.GetComponent();
        animator.runtimeAnimatorController = avatarController;
    }

    public void StartWalkAnimation()
    {
        this.animator.SetBool("isMoving", true);
    }

    public void StopWalkAnimation()
    {
        this.animator.SetBool("isMoving", false);
    }

    public void StartTurnAnimation(float angle)
    {
        if (angle > 0)
        {
            this.animator.SetBool("isTurningRight", true);
        }
        else
        {
            this.animator.SetBool("isTurningLeft", true);
        }
    }

    public void StopTurnAnimation()
    {
        this.animator.SetBool("isTurningLeft", false);
        this.animator.SetBool("isTurningRight", false);
    }

    public bool IsTurnAnimatorPlaying()
    {
        return animator.GetCurrentAnimatorStateInfo(0).IsName("RightTurn") || animator.GetCurrentAnimatorStateInfo(0).IsName("LeftTurn");
    }

    public bool IsMoveAnimatorPlaying()
    {
        return animator.GetCurrentAnimatorStateInfo(0).IsName("Walking");
    }

    private void OnDisable()
    {
        avatarImporter.OnAvatarStored.RemoveListener(AssignAvatarController);
    }
}

5.5 Movement Controller :

Create a new script named MovementController and copy the following code. The code will listen to the Unity event declared in the ARTapToPlace script. When the event gets invoked, the avatar is made to rotate and move it to the new position.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class MovementController : MonoBehaviour
{
    [SerializeField] private AvatarImporter avatarImporter;
    [SerializeField] private ARTapToPlace arTapToPlace;
    [SerializeField] private AvatarAnimationController avatarAnimationController;

    private Vector3 touchPos;
    private Transform avatarTransform = null;

    private bool isTurnning;
    private bool isMoving;

    private void OnEnable()
    {
        arTapToPlace.OnNewTouch.AddListener(OnTouch);
        avatarImporter.OnAvatarStored.AddListener(GetAvatarTransform);       
    }

    void Update()
    {
        if (isTurnning)
        {
            if (Vector3.Dot(avatarTransform.forward, Vector3.Normalize(touchPos - avatarTransform.position)) > 0.9f)
            {
                avatarAnimationController.StopTurnAnimation();
                isMoving = true;
                isTurnning = false;
            }
        }

        if (isMoving && !avatarAnimationController.IsTurnAnimatorPlaying())
        {
            if (Vector3.Distance(avatarTransform.position, touchPos) > 0.1f)
            {
                avatarTransform.LookAt(touchPos); // to account of that 0.1 rotation that's missed

                if (!avatarAnimationController.IsMoveAnimatorPlaying())
                    avatarAnimationController.StartWalkAnimation();

                // it's normalized so that the velocity remains constant. 
                // 0.3 is a magic number to match the speed with the animation speed
                avatarTransform.position += (touchPos - avatarTransform.position).normalized * 0.3f * Time.deltaTime;

                //  the below code works as well
                //  avatarTransform.position = Vector3.MoveTowards(avatarTransform.position, touchPos,  0.3f * Time.deltaTime);
            }
            else
            {
                avatarAnimationController.StopWalkAnimation();
                isMoving = false;
            }
        }

    }

    private float CalculateAngle(Vector3 targetPos, Vector3 currentPos, Transform avatarTransform)
    {
        float angle = Vector3.SignedAngle(currentPos - targetPos, avatarTransform.forward, avatarTransform.up);
        return angle;
    }

    private void OnTouch()
    {
        avatarAnimationController.StopWalkAnimation();
        avatarAnimationController.StopTurnAnimation();
        touchPos = arTapToPlace.TouchPosition;

        //using the helper method to calculate the angle
        float angle = CalculateAngle(touchPos, avatarTransform.position, avatarTransform);

        if (angle == 0)
        {
            isTurnning = false;
            isMoving = true;
        }
        else
        {
            isTurnning = true;
            isMoving = false;
            avatarAnimationController.StartTurnAnimation(angle);
        }
    }

    private void GetAvatarTransform()
    {
        avatarTransform = avatarImporter.ImportedAvatar.transform;
    }

    private void OnDisable()
    {
        arTapToPlace.OnNewTouch.RemoveListener(OnTouch);
        avatarImporter.OnAvatarStored.AddListener(GetAvatarTransform);
    }
}

Angle calculation Explained:

• The API SignedAngle will return the smaller of the two possible angles between the two vectors, therefore the result will never be greater than 180 degrees or smaller than -180 degrees.
• The SignedAngle method requires 2 vectors to calculate the angular difference and a third vector around which the other vectors are rotated.
• Legend for the image below :
• Vector A is Avatar's forward vector
• Vector B is the vector joining Avatar’s position to Touch position 1
• Vector C is the vector joining Avatar’s position to Touch position 2.

Case 1: Considering touch position 1
For this case, the angle will be calculated between Vector A and Vector B in a clockwise direction as that’s the shortest path. So the returned value will be positive between 0 and +180.

Case2:
For this case, the angle will be calculated between Vector A and Vector C in a clockwise direction as that’s the shortest path. So the returned value will be positive between 0 and -180.

Dot product for rotation explained

• When the rotation animation gets played the whole avatar rotates and so does its local transform.
• If the dot product of two normalized vectors is:
• 1 then, the vectors are in the same direction.
• -1 then, the vectors are in opposite direction.
• 0 then, the vectors are 90deg to each other.
• Legend for the image below :
• Vector A is Avatar's forward vector
• Vector B is the vector joining Avatar’s position to Touch position.
• Vector C is the Normalized vector joining Avatar’s position to Touch position.
• As the avatar rotates, the avatar’s forward vector changes and so does the dot product.

6. Adding Components

Let’s now add the components we have created to get the desired result.

• Add the ARPlaneDetection component to the AR Session origin GameObject→ drag and drop the required fields i.e the AR Plane Manager and the Floor prefab.

• Add the ARTapToPlace component to the AR Session origin GameObject → drag and drop the ImportAvatar and the AR Camera components in the respective fields.

• Add the AvatarAnimationController component to the AR Session origin GameObject → drag and drop the required fields i.e the ImportAvatar and the AvatarController we had created earlier

• Finally, add the MovementController component to the AR Session origin GameObject → drag and drop the ImportAvatar, the ARTapToPlace and the AvatarAnimationController into the respective fields.

With that, we have completed creating our scene. All that’s remaining now is to test it and see if it works as intended.
Ps: It will work correctly, I have already tested it and made sure all the above code is correct... haha.

7. Testing

To test the app,

• Connect your phone to your laptop/ PC. Also, make sure USB Debugging is enabled on your phone.
• Navigate to File → Build Settings → click on Add Open Scenes → click on Build and Run -> click on Continue with 'demo' subdomain.

Wait till the build gets completed and the app will automatically be launched on your phone. Now you can test it by tapping at a different location on the floor and by tapping outside the floor area. While testing, you can see that the avatar moves to the location of touch on the floor. If you touch anywhere apart from the floor the avatar will not move.

Conclusion

In this blog, we saw how we can use the imported Avatar for AR. So what next? Well, what we saw was just one application the Ready Player Me Android integration. There are many other ways you can make use of the avatar as well, for example, you can use it as your main character in a game.

You can try out different things in AR as well. You can have a joystick to control your Ready Player Me Avatar in AR, make it run, jump, etc.

‍