Spatial Mapping

In this tutorial, you will learn how to build a procedural mesh of your room and create applications with real world navigation and physics.

Introduction

Spatial mapping captures a mesh of your environment, allowing physical interactions between the real and virtual world. For example, you can use Unity’s runtime navigation system to bake the mesh into a “navigation mesh” (or “NavMesh” for short) which tells characters where they can walk/move so they can intelligently traverse around the real-world.

It also allows you to use Unity’s built-in physics engine, adding convincing physical behaviour to virtual objects. This way, they can collide with the real world, bounce off of it in appropriate directions, accelerate correctly and be affected by gravity and other forces present in the real world.

Create a new Project

  • Select File > New Project and click Create project.
  • Make sure the ZED SDK is installed, and import ZEDCamera.unitypackage.
  • Select File > New Scene.
  • In the Hierarchy panel, delete the Main Camera.
  • Drag the ZED_Rig_Mono.prefab to the Hierarchy.

Mapping an Area

We will now add a Spatial Mapping script to allow meshing of the real-world environment. A procedural mesh will be generated which is a type of mesh geometry that can be created and modified at runtime in Unity.

  • Drag to the Hierarchy the ZED_Spatial_Mapping prefab located in ZED > Examples > SpatialMapping > Prefabs. It is an empty GameObject with an attached script ZEDSpatialMappingManager.cs.

  • Click on ZED_Spatial_Mapping in the Hierarchy to display the Spatial Mapping panel.

For more details on Spatial Mapping parameters, read this section. We recommend using the following set of parameters:

  • Resolution = MEDIUM
  • Range = MEDIUM
  • Filtering = LOW

Now, let’s start mapping the area.

  • Launch the scene

  • Press the button Start Spatial Mapping to launch the mapping. Now move the ZED around to capture a mesh of your environment.

  • Click on Stop Spatial Mapping to stop updating the mesh and start mesh filtering.

  • Display the final mesh by pressing the button Display Mesh.

Mesh Holder

In the Hierarchy, you will now see a Mesh Holder object storing the mesh of the real-world environment. If you didn’t select the Save Mesh option, the mesh will be temporarily stored in RAM and deleted at the end of the session.

Expand the mesh holder to see the different chunks (submeshes) that compose the global mesh. During procedural meshing, the surrounding area is split up into regions. Each region corresponds to a chunk. When spatial mapping is stopped, these chunks are merged to create larger submeshes. The submeshes have a maximum vertex count of 65535, which is the maximum number of vertices Unity can handle per mesh. The chunks are used to generate the NavMesh and handle complex collisions with the real world.

Using the Navigation Mesh

To add a character that navigates the real world, we will use Unity’s navigation system. Unity can create a Navigation Mesh which is a simplified plane that describes the walkable surfaces of a game. This allows a character to automatically find its path through obstacles.

As of version 5.6, Unity supports Runtime NavMesh building, which allows to automatically bake NavMesh at runtime using the environment geometry.

Building the NavMesh

The NavMeshSurface.cs component, which is already attached to your ZED_Spatial_Mapping prefab, builds a NavMesh Surface at runtime. A NavMesh Surface represents the walkable area for one Agent type. Multiple NavMesh surfaces can be present in a scene. Once mapping is done, the NavMeshSurface component will collect all chunk sources and build a NavMesh for a given Agent Type.

  • Select your ZED_Spatial_Mapping prefab and, in the Inspector, look for the Nav Mesh Surface component.

  • Select your Agent Type.

  • Launch Spatial Mapping and map the area.

  • Press Stop Spatial Mapping. The NavMesh will be generated from the procedural mesh.

  • Toggle between DisplayMesh and Hide Mesh to show/hide the NavMesh.

Note: It is not possible to bake a NavMesh at runtime using Window > Navigation > Bake as one would with imported 3D meshes. You need to bake the NavMesh using the scripts provided or Unity Components for navigation.

Adding Agents

Let’s place a Bunny character in the scene and make it move in random patterns on the walkable surface.

A NavMeshSurface is required for each type of agent, and an agent of one type cannot use surfaces of other types. This is because the navmesh defines the agent’s mobility around the scene, and different agents have different mobility: taller enemies can step over bigger objects, flying enemies can ignore terrain, etc. The NavMeshSurface component has a drop-down where you select which agent type to bake for.

  • Select Humanoid Agent Type.
  • Drag the ZomBunny prefab located in Examples > SpatialMapping > Resources to the Hierarchy.
  • To automatically place the Bunny on the generated NavMesh, add the NavMeshAgentController component on your agent. The script sets a random starting position for the character on the NavMesh.
  • Create an empty GameObject and attach the EnemyManager component. The script makes the Bunny move in random patterns on the walkable surface determined by the NavMesh.
  • Play the scene, start mapping a large area and when you press Stop Spatial Mapping, the NavMesh is generated and the Bunny character appears in the scene.

Using Physics

At the end of spatial mapping, the mesh is automatically assigned a component MeshCollider to resolve collisions. We can now throw virtual balls in the real world and let Unity handle physics and collisions with the underlying mesh.

  • Add the script BallLauncher located in Examples > SpatialMapping > Scripts to the Camera_Left GameObject.
  • Select a ball type with the prefab located in Examples > SpatialMapping > Resources > Physics > Sphere.prefab
  • Play the scene and start throwing bouncing balls in the real-world environment!