Green Screen VR Capture Capturing VR in Mixed reality In this tutorial, you will learn how to capture VR applications in mixed-reality using the ZED, an HTC Vive or Oculus Rift and a green screen. For an in-depth article on capuring VR content with an external camera, read this tutorial. The main advantage of using depth data from the ZED camera is pixel occlusion and lighting. This means that the depth of every pixel in the camera’s view will be compared against its corresponding virtual pixel, so the nearest one will be rendered. In addition, you can enhance your real scene with virtual lights and shadows. Requirements For this tutorial, you will need the following items: ZED camera Green screen HTC Vive headset or Oculus Rift 2x Vive controllers (or 1x Vive tracker and 2x controller) or 2x Oculus touch. First, you will need to mount a controller/tracker onto the ZED and make sure they are solidly attached. The controller will let you track the ZED in the same space as the HMD and allow you to move the camera while filming. To attach the ZED and controller/tracker, you can download and 3D print the support available here. If you don’t have access to a 3D printer, feel free to improvise with clamps, tape, etc. But you must ensure two things: First, that your attachment mechanism doesn’t occlude too many of the Tracker’s sensors. Second, your attachment must be firm (no wobbling) or else you will have to recalibrate constantly. How It Works To start recording mixed reality footage, we will go through the following steps: Create a new Unity project and import the ZED_Green_Screen prefab Adjust ZED camera settings to improve image quality Adjust chroma key to remove the green screen Add the headset Note: There is a known issue where using the Vive and ZED together in Unity can freeze the program. The issue has been fixed with the latest Nvidia drivers. Read more. Create a new Project Open your Unity project, or create a new project. Follow the instructions here on importing the ZED Unity package. In Edit > Project settings > Player, check Virtual Reality Supported. In the Project panel, navigate to Asset > ZED > Examples > GreenScreen > Prefabs. Drag and drop the ZED_GreenScreen.prefab in the Hierarchy view. In the Hierarchy view, select ZED_ GreenScreen > Camera_Left. Click on the Play button. Your Unity interface should look like this: Adjusting Camera Parameters In the ZED_ GreenScreen > ZED Manager (Script), make sure the camera is set in HD1080 mode (1920*1080 resolution). Changing the resolution requires to restart the scene. After selecting ZED_ GreenScreen > Camera_Left, click on the Camera Control button of the Green Screen Manager in the Inspector view. By default, the camera exposure and white balance are adjusted automatically. Auto White Balance tends to add a green tint to the picture when placing the ZED in front of a green screen. Since you’re shooting in a controlled environment with fixed lighting, you can disable the Automatic settings and manually adjust the parameters of the Camera Control panel. Hit Save once you’re happy with your settings, so they can be easily loaded later after restarting Unity. Adjusting Chroma Key Settings In the Inspector panel, you can adjust your chroma key settings with the Green Screen Manager script. Below are explanations of what each setting does and when to adjust it. Views Select the FOREGROUND view to get a better feel for what remains visible in your scene and what doesn’t. This can be useful for setting the critical settings in the next section. Change the view to FINAL to see the result. Color Range The Color, Range and Smoothness values are the most important and where you’ll likely spend the most time adjusting. When the script goes through each frame, it will check each pixel’s color to see if it should turn it invisible. If you’re using a background that’s not green, adjust the Color setting to match. Otherwise, adjust the shade of green to better fit the screen’s exact color and your lighting setup. Range indicates how close a color on the spectrum can be to the exact color of your background. You need to add some range because even the highest quality green screen studio has shading to it. Smoothness defines how colors just outside the range get blended in. With zero smoothness, a color just outside the range is completely visible. With more smoothness, it will be partially visible. You’ll get further control over how smoothness effects the image in the next section. The easiest way to find a good Range and Smoothness is to start the scene, set both values to 0, adjust the Range, then adjust the Smoothness. Refining the matte Next, refine the edge of the matte with the Erosion, Black Clip, White Clip, Edge Softness and Despill settings. The ALPHA view can be helpful for making these adjustments. The Erosion setting erodes the edges of the matte to remove the remaining green areas. Without erosion With erosion White Clip and Black Clip: Alpha values above the White Clip value will be set to 1 (fully opaque), and alpha values below the Black Clip value will be set to 0 (fully transparent). In other words, White Clip makes nearly opaque pixels fully opaque, and Black Clip does the same for nearly invisible pixels. This helps you remove noise that results from your smoothness setting. Without white clip With white clip Despill reduces the green screen color reflections on the key. Increase this if you have a slight green hue showing on your subject, but be careful not to use more than you need or the other colors will appear less vibrant. Change the view to FINAL before adjusting color despill. Without despill With despill Finally, store your chroma key parameters by clicking on Save. Garbage Matte The Garbage matte script lets you extend the virtual world beyond the green screen’s limited size. It allows you to define a region in space and exclude image pixels that do not belong to this region. To use camera tracking with the garbage matte (as we don’t do in this demo as we’re using the headset) you need to activate tracking before setting your garbage matte. Tracking can be activated by clicking on the ZED_Greenscreen prefab, and enabling it in the ZED Manager compponent. Click on Enable Garbage Matte Start your scene. Make sure Place Markers is selected. Define your green screen region by clicking on the image in your Game display window to place your markers. You should see spheres appear at each place you click. Repeat the previous step until the whole green screen area is covered with a white mesh. Click on Apply Garbage Matte to exclude the outside region of the mesh (set to transparent). To save and load markers, use the Save/Load option. Note: If the area is not covered correctly, use Remove Last Marker to disable the garbage matte and remove the last marker. You can also adjust the position of a specific marker with a right click to select a sphere and a left click on a new position. Adding the VR headset Interfacing the ZED with the Vive Download and import the SteamVR Unity package. Add the [CameraRig] prefab located in the folder SteamVR / Prefabs. Right click in the Hierarchy view, and then on Create Empty. Drag the ZED_GreenScreen prefab and drop it on the GameObject. Launch the sample scene with the Play button. The GameObject ZED_GreenScreen contains the script ZEDSteamVRControllerManager. It loads the calibration configuration created by the calibration software if you choose to use it (beta). It also adds a controllable delay to the movement of the virtual controllers, which is important as the difference in latency between the controllers and ZED’s output cause the real and virtual controllers to fall out of alignment if moved quickly. Note: Oculus Rift is also supported through SteamVR plugin. Manual Calibration Do this step only if you prefer not to use the beta calibration software and ZEDSteamVRControllerManager. To set up your calibration manually, disable the script ZEDSteamVRControllerManager and add a SteamVR_Tracked_Object to the parent. In these script parameters, you can select the controller attached to the camera in the Index list. Finally, manually adjust the transform of the ZEDGreenScreen to get a matching physical and virtual world. Put a second Vive controller in front of the ZED. If the system is not calibrated, you should see that the real and virtual controllers do not overlap. Click on ZED_GreenScreen and adjust the Transform values till the position and rotation of the virtual and real controller match. Try to put the controller closer to the camera and adjust translation values, then move further away and adjust rotation. You may need to repeat the previous steps several times to achieve a good calibration of the real and virtual world. Also try moving the controller to both edges of the screen. To save your configuration, click on Save Camera Offset in the Inspector view > ZED Position Manager. This will save a ZED_Position_Offset.conf file. In the latest SDK update, we provide a semi-automated calibration app that lets you create the offset file automatically. You can download a beta version here.