Microsoft HoloLens By Example

Joshua Newnham

• Publisher: Packt Publishing
• Year: 2017
• eBook ISBN: 9781787122369
• Edition: 1
• Topic: Informatik
• Subtopic: Programmierung in C#

Get to grips with HoloLens development as you create mixed reality apps from scratchAbout This Book• Create awesome Augmented Reality (AR) apps for the Microsoft HoloLens platform• Unleash the power of Unity SDK for HoloLens to create next generation AR apps• Explore the exciting world of gesture control, visual mapping, voice command for apps, and many more cutting edge possibilities with HoloLensWho This Book Is ForThis book is for developers who have some experience with programming in any of the major languages such as C#, C++, and so on. You do need any knowledge of Augmented Reality development.What You Will Learn• Extend the computing experience beyond the flat glass screen by placing and embedding virtual objects holograms) into the physical world• Interact with the holograms using gaze, gestures, and voice• Enhance the experience with spatial sound• Allow multiple users to naturally collaborate with each other• Integrate voice commands into your own HoloLens projects• Experiment with techniques to better understand the real world• Implement a user interface in Mixed Reality• Blend the virtual and physical world by making the holograms interact and react to the physical environmentIn DetailAre you a developer who is fascinated with Microsoft HoloLens and its capabilities? Do you want to learn the intricacies of working with the HoloLens SDK and create your own apps? If so, this is the book for you.This book introduces and demystifies the HoloLens platform and introduces new ways you can interact with computers (Mixed Reality). It will teach you the important concepts, get you excited about the possibilities, and give you the tools to continue exploring and experimenting. You will go through the journey of creating four independent examples throughout the book, two using DirectX and two using Unity. You will learn to implement spatial mapping and gesture control, incorporate spatial sound, and work with different types of input and gaze. You will also learn to use the Unity 5 SDK for HoloLens and create apps with it.Collectively, the apps explore the major concepts of HoloLens, but each app is independent, giving you the flexibility to choose where to start (and end).Style and approachThis book takes an example-based approach where you'll build AR apps with increasing difficulty. You will learn more about HoloLens platform as well as AR app development in general.

Assistant Item Finder Using DirectX

In the previous chapter, we explored the opportunities that HoloLens opens up by being able to see everything you see; our exploration began by setting up and walking through the holographic DirectX 11 application template Microsoft has bundled in Visual Studio 2015 Update 3. We then walked through extending this application to detect and identify faces, and display associated metadata about the identified person the user was looking at.

In this chapter, we continue our exploration with how HoloLens can better assist people through a proof of concept example that continuously observes the world, identifying and tracking salient items that you (it) see. If the user happens to forget where an item is, they will be able to ask for assistance and be guided via a breadcrumb trail from the user to the item. For instance, imagine, if you will, a time when you have misplaced your keys and spent vast amounts of time searching for them. Now, imagine a world, in the very near future, where your device can conveniently track your items and lead you to them--possibly adapting the path that best suits your needs, for example, by that is routing you through a wheel chair-friendly path.

Once again, I encourage you to let your imagination wander while you read this chapter, dreaming of how this concept can be extended. One example might involve, identifying what is in the fridge and automatically constructing and presenting a shopping list when the user is in proximity of a grocery store. These are exciting times ahead. With that being said, let's get started!

We will start by introducing the application and all of the major components, then walk through each one until we have a functioning prototype. We begin by mapping out the environment, then tagging the location of the items we detect, and finally, building and rendering the trail from the user to the "lost" item. In this chapter, you will learn the following topics:

• Creating and using SpatialLocatorAttachedFrameOfReference to consider the user's location from the perspective of the device
• Making use of spatial anchors to map the physical world
• Using Microsoft Cognitive Services to recognize objects

Project setup

Our first stop will be looking at how we can make sense of the world, or at least, programmatically be aware of what the user is looking at. Computer vision, specifically recognition, has made leaps and bounds since 2012, when computer scientists Geoffrey Hinton, Alex Krizhevsky, and Ilya Sutskever entered the ILSVRC 2012 computer vision competition using ideas from http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf, a paper they had recently published. Being the only ones using a Convolutional Neural Network (CNN), they entered the competition; the rest is pretty much history. Models these days can compete with humans in recognizing objects in images.

CNN is a type of neural network well suited for images due to its properties of preserving the relationship between pixels in close proximity.

Fortunately for us, many companies have made Application Program Interfaces (APIs) available that offer similar capabilities for computer vision, including Microsoft. We will be using Microsoft's Cognitive Services Computer Vision API for this example. The API gives our application the ability to recognize objects in a image, or in our case, in front of the user. We will use this to track what items the user comes across. Before jumping into code, let's see how this API works and what capabilities it provides for us.

The easiest way to do this is to go to their service webpage https://azure.microsoft.com/en-gb/services/cognitive-services/computer-vision/ and explore some of the examples presented on this page. The following is one such example showing the image and associated metadata:

Screenshot of the computer vision Microsoft offers as ones of its cognitive services, Source: https://azure.microsoft.com/en-gb/services/cognitive-services/computer-vision

In the preceding image, you can see the service returning a set of Tags and Captions. This is what we will assign, each frame (captured image) we capture and we will use this data when searching for an item for the user.

Adding the Computer Vision API service

As we did in the previous chapter, to use this service, we will need to add it to our Microsoft Azure portal. Head over to https://portal.azure.com/ on your browser and log in (or register if you don't already have an account).

Once logged in, you will be taken to your Dashboard; from here, click on the +New button at the top of the left panel. This will expand out a menu. Within this menu, search and click on AI + Cognitive Services, which will spawn another menu where we can select the Computer Vision API, as shown in the following screenshot:

After clicking on the Computer Vision API item, you will be presented with a tab requiring you to enter the details of your service, such as Name, Subscription, Location, Pricing tier, and Resource Group, and help is accessible within the user interface. With all that done, you have your service up and running; details of this service should be visible on the screen and can be accessed from your dashboard, as shown in the following screenshot:

To use your service, you will need to call it using the generated keys. These can be accessed by clicking on Show access keys... which, when clicked, will display a set of keys you will need. For convenience, copy and paste them somewhere easily accessible (Notepad) for use later in this chapter.

The project

To recap, the objectives for this application are as follows:

• Plot out the environment so we can reconstruct a path from the user's location to the specified item
• Continuously process frames captured by the HoloLens color camera and attach the tags returned by Microsoft's Cognitive Service Vision API

But first, let's set up the environment and then look at how we can better understand the environment by plotting paths where the user walks using spatial anchors (via SpatialAnchor). Start by launching the starter solution for this chapter; this can be found in the directory Chapter3/Starter from the repository https://github.com/PacktPublishing/Microsoft-HoloLens-By-Example. Launch the Visual Studio solution by either double-clicking on the solution file AssistantItemFinder.sln or by loading from within Visual Studio by navigating via File | Open | Project | Solution and selecting the AssistantItemFinder.sln file. The starter solution is more or less the standard Visual C# holographic DirectX 11 app template that we used in the previous chapter, with the addition of some classes and resources used to render the trail for the user to follow.

Before we start coding, let's get some administration out of the way. The first is to set the appropriate Capabilities for this project, which is access to the color camera (Webcam) and Internet (Internet (Client)). Open up the Capabilities panel of the package properties by double-clicking on the Package.appxmanifest file and selecting the Capabilities tab. Once open, check the capabilities:

• Internet (Client)
• Webcam

Your Capabilities panel should look similar to the screenshot with the previously mentioned capabilities checked:

Next, we will amend the cognitive_service.key file that resides in your projects directory with the key from the service we just created. If you have not done so already, copy the key from the service we created earlier and paste it into this file. We use this when we build the class responsible for calling the Comp...

Table of contents

• Title Page
• Copyright
• Credits
• About the Author
• About the Reviewer
• www.PacktPub.com
• Customer Feedback
• Preface
• Enhancing Reality
• Tagging the World Using DirectX
• Assistant Item Finder Using DirectX
• Building Paper Trash Ball in Unity
• Building Paper Trash Ball Using Holotoolkit in Unity
• Interacting with Holograms Using Unity
• Collaboration with HoloLens Using Unity
• Developing a Multiplayer Game Using Unity
• Deploying Your Apps to the HoloLens Device and Emulator