Motion Controllers in AR

In this demo, we’ll make your right hand carry around a 3D cube, and turn your left hand into a flashlight.

This tutorial assumes you’ve finished our Getting Started guide. You can use either an Oculus Rift or HTC Vive. You will have to set up your headset’s tracking base stations as, unlike the ZED Mini on your headset, the controllers have no means of tracking their position.

Starting Your Project

  • Start a new scene, import the ZED plugin, delete the MainCamera and drag ZED_Rig_Stereo into the Hierarchy.

  • With ZED_Rig_Stereo selected, go to the inspector and uncheck Enable Tracking. This will cause the ZED to rely on the Rift/Vive’s tracking, which will keep it better synchronized with the controller positions.

Tracking the Controllers

We’ll use free plugins to handle the controller positions. This is different whether you are using the Oculus platform (Oculus Rift only) or SteamVR (Vive or Oculus Rift). Follow only the relevant subsection.

Oculus Integration

  • If you don’t already have an Asset Store tab in your editor, open a new one by going to Window -> Asset Store, or hitting Control-9 for short.
  • In the store, search for “Oculus” and look for “Oculus Integration.” Once on that page, click Import.

  • After a moment you’ll see a list of the files and folders in the plugin, prompting you to choose which to import. You can leave them all checked. Click Import.

Note: There are many assets you don’t need that you can choose not to import, but it’s easy to accidentally uncheck something important if you’re not careful. However, you can safely uncheck the OVR Avatar folder to drastically reduce the import time, assuming you don’t have plans for it.

  • In your Assets tab, go into OVR -> Prefab and drag an OVRCameraRig into your Hierarchy. Then click the arrow beside it to expand it.

  • Select LeftEyeAnchor, CenterEyeAnchor, and RightEyeAnchor, and in the Inspector, disable them. We do this so that they do not override the ZED_Rig_Stereo’s cameras and to save performance.

The LeftHandAnchor and RightHandAnchor are the virtual representations of your controllers. Anything you want to add to the controller should either be added directly to these objects, or to objects parented to them.

SteamVR Plugin

  • If you don’t already have an Asset Store tab in your editor, open a new one by going to Window -> Asset Store, or hitting Control-9 for short. In the store, search for “SteamVR” and look for “SteamVR Plugin.” Once on that page, click Import.

  • You’ll get a prompt that you’re importing a complete project. Unless working in an existing project with custom settings, you can safely click Import.

  • You’ll get a list of files and folders you can choose to import or not. It’s simplest to leave them all checked and click Import.

  • Create an Empty GameObject by right-clicking in the Hierarchy with nothing selected and clicking Create Empty. Then rename it “Right Hand.” Do this a second time, and rename the second one “Left Hand.” These objects will represent your controllers’ positions and locations in the virtual space.

  • To make them move with your controllers, select both objects, and in the Inspector, add the SteamVR_TrackedObject component.

Note: You may notice there’s a SteamVR_TrackedController script, which sounds more appropriate. But this script has other behaviors designed for VR that can interfere with our app.

  • With Right Hand selected, set the SteamVR_TrackedObject’s Index to Device 1. Then set the same value in Left Hand to Device 2. These will set them to follow the positions/rotations of the first and second devices shown in your SteamVR overlay after the HMD, which should be your right and left controllers.

Note: Your left and right controllers are typically represented as device 1 and 2, but the assignment can be somewhat arbitary depending on the order your devices was turned on. If the objects position themselves at your base stations instead of your hands, either change the indexes or add additional implementation to find the left/right controller indexes at runtime.

Augmenting Your Hands

Unlike VR, it’s not recommended to put an object in your user’s actual hand, such as a replica of their controller. Doing so makes any tracking errors extremely noticeable, since they have a near-perfect frame of reference, and right in front of their face.

Instead, we’ll add an object floating in front of one hand, and make the other hand a flashlight without adding a visible model at all.

Add a Floating Cube

  • Right-click your right hand object (we called it “RightHandAnchor” with Oculus and “Right Hand” in SteamVR) and add a cube to it as a child by clicking 3D Object -> Add Cube.

  • With the new cube selected, go to the Inspector and change its Z Position to 0.2 and its Scale to X = 0.1, Y = 0.1, Z = 0.1. This will make it hover 20 centimeters in front of the controller and shrink it down to a manageable size.

Adding a Flashlight

Unlike the cube, we don’t need to represent the light as a visible 3D object if we don’t want to. Instead, we’ll add the proper components directly to the left hand object.

  • Select your left hand object (“LeftHandAnchor” or “Left Hand”). In the Inspector, add the ZEDLight script. When you add it, it will automatically add Unity’s Light component as well.

Note: If using SteamVR, in addition to the components pictured above, you’ll also see the SteamVR_TrackedObject component you added earlier.

  • Change the Light component’s Type to Spot, the Color to a bright color of your choosing, and the Intensity to 10. This’ll make it a disco-style spotlight attached to your hand.

Run the Scene

You’re ready to go! Run the scene, grab your controllers and play around. Move the cube around and behind real-world objects. Shine the light at surfaces near and far away, and with different angles. Change the color of the light to see how it looks. Congrats on becoming an augmented human being!