Learn ARCore - Fundamentals of Google ARCore

Micheal Lanham

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781788833639
• Edition: 1
• Topic: Computer Science
• Subtopic: Web Development
• Index: Computer Science

Create next-generation Augmented Reality and Mixed Reality apps with the latest version of Google ARCore

Key Features

• Harness the power of the Google's new augmented reality (AR) platform ARCore to build cutting-edge Augmented reality apps
• Learn core concepts of Environmental Understanding, Immersive Computing, and Motion Tracking with ARCore
• Extend your application by combining ARCore with OpenGL, Machine Learning and more.

Book Description

Are you a mobile developer or web developer who wants to create immersive and cool Augmented Reality apps with the latest Google ARCore platform? If so, this book will help you jump right into developing with ARCore and will help you create a step by step AR app easily.

This book will teach you how to implement the core features of ARCore starting from the fundamentals of 3D rendering to more advanced concepts such as lighting, shaders, Machine Learning, and others.

We'll begin with the basics of building a project on three platforms: web, Android, and Unity. Next, we'll go through the ARCore concepts of motion tracking, environmental understanding, and light estimation. For each core concept, you'll work on a practical project to use and extend the ARCore feature, from learning the basics of 3D rendering and lighting to exploring more advanced concepts.

You'll write custom shaders to light virtual objects in AR, then build a neural network to recognize the environment and explore even grander applications by using ARCore in mixed reality. At the end of the book, you'll see how to implement motion tracking and environment learning, create animations and sounds, generate virtual characters, and simulate them on your screen.

What you will learn

• Build and deploy your Augmented Reality app to the Android, Web, and Unity platforms
• Implement ARCore to identify and visualize objects as point clouds, planes, surfaces, and/or meshes
• Explore advanced concepts of environmental understanding using Google ARCore and OpenGL ES with Java
• Create light levels from ARCore and create a C# script to watch and propagate lighting changes in a scene
• Develop graphics shaders that react to changes in lighting and map the environment to place objects in Unity/C#
• Integrate motion tracking with the Web ARCore API and Google Street View to create a combined AR/VR experience

Who this book is for

This book is for web and mobile developers who have broad programming knowledge on Java or JavaScript or C# and want to develop Augmented Reality applications with Google ArCore. To follow this book no prior experience with AR development, 3D, or 3D math experience is needed.

Recognizing the Environment

Throughout this book, we have looked at the numerous ways of how our device, with the help of ARCore, can track the user, understand the user's world, and render an alternate reality. ARCore uses the device's sensors and camera as inputs to constantly update what it perceives as the user's real world. However, what if we wanted to do more for the user; perhaps identify a certain object, sign, or landmark? That would require a much more advanced set of tools. Even just 5 years ago, this would seem like an incredibly daunting task. With the advent of OpenAI, thanks to Mr. Musk, many other companies have started to open source and make their tools available. This has led to phenomenal explosive growth in these technologies, colloquially referred to as Machine Learning (ML), and broadened their accessibility to everyone. Fortunately, for those interested in developing AR apps, this is a good thing. We want all the help we can get when it comes to recognizing and understanding the user's environment.

For this chapter, we will introduce ML and explore how we can use it to create better AR apps for our users. In this chapter, we will cover the following topics:

• Introduction to ML
• Deep reinforcement learning
• Programming a neural network
• Training a neural network
• TensorFlow

Machine Learning is a very advanced subject that can take years of study in order to master. However, for our purposes, we will learn some basic techniques, which the reader can extend on later, either through more learning or implementing their own solution.

If you already have an in-depth understanding of neural networks, convolutional neural networks, and TensorFlow, feel free to breeze over this chapter.

Introduction to ML

Machine Learning is a term widely used to refer to artificial intelligence and related computer predictive analytical models. The name Machine Learning, while perhaps overly generalized, fits better than the term AI. However, Machine Learning is itself such a broad term that it perhaps needs some further explanation and clarification. A machine obviously refers to a computer, or other device and learning tends to denote an algorithm or model that will evolve or learn over time. However, this is often not the case in many Machine Learning models. Therefore, for our purposes, we will use the broader term of Machine Learning to refer to any tool or algorithm that can be trained to recognize the environment or parts of the environment in AR, thus allowing us, the developers, to better augment our user's world.

Data science and Machine Learning go hand in hand. Data science is all about making sense of data, extracting patterns, and making predictions. In essence, when you start writing Machine Learning models in order to recognize objects or the environment, you are really just analyzing data, which means you can also, very loosely, call yourself a data scientist.

Machine Learning is a big area and is only getting bigger every day, so let's break down the specific problems we would like ML to help us with:

• Target detection: Targets have been used in AR for some time. It has been the primary tracking and reference point for many AR apps previous to ARCore.
• Image recognition: This spawns into a whole set of sub-applications, all of which we will deal with in detail later.
• Object detection: Being able to detect an object in 3D from point cloud data is no easy feat, but it has been done and is getting better.
• Face detection: Detecting a person's face in an image has been around for years and has been used to great effect in many apps.
• Person detection: Detecting people or motion has great possibilities. Think Kinect comes to AR.
• Hand/Gesture detection: Not to be confused with touch gestures. This is where we detect a user's hand motions or gestures in front of a device's camera.
• Pose detection on object: Related to object detection, but now we also detect the position and orientation of the object.
• Light source detection: Being able to place realistic lights in a scene to make virtual object rendering more realistic. We already looked at the importance of lighting in Chapter 7, Light Estimation.
• Environment detection: Recognizing the environment a user has moved into has great application in mapping buildings or other locations where GPS is unavailable, which applies to most internal spaces.

Each of those problems may require different tools and techniques to solve those issues. In ML, it's not always about using the tool but the final answer and what works. Think about this as you build any ML you need for your app. Try a variety of ML tools and techniques; differences in size and performance of ML models can be critical, and it's something you need to consider.

A Machine Learning algorithm walks into a restaurant.
The waiter asks, "What will you have?
The algorithm says, "What's everyone else having?"
- Unknown

In the following table is a summary of the current major ML providers and the types of AR problems they can be used to solve:

Toolset	Pros/Cons	Machine Learning task
		Targets/Image	Object/Pose	Face	Person	Hand	Light	Environment
Vuforia	Mature and easy to use. Requires internet connectivity.	Yes	Yes/Paid
XZIMG	Face and image/target tracking supported for Unity and other platforms.	Yes		Yes
ARToolkit	Mature OpenSource platform for image tacking and feature detection.	Yes
EasyAR	Pro license gets object and feature tracking.	Yes	Yes/Paid
Google Face Detection API	Low level Android API.			Yes
OpenCV	A mature low-level API for Android, commercial version ported to Unity. Still requires low level knowledge.	Yes	Yes	Yes	Yes	Yes	Coming	Coming
Google TensorFlow	Still in its infancy but quickly becoming the platform standard for CNN. Low level and advanced ML knowledge required.	Yes	Yes	Yes	Yes	Yes	coming	coming
Google ARCore	Currently, identifies planes, feature points, and light.					Yes	Yes

We only included the main players who have built an AR platform for a mobile ARCore-supported device. Web technologies were omitted from this due to their limitations, although many of the mentioned technologies require internet connectivity and support web platforms as well. If you quickly review the table, you can also clearly see two main contenders that have the potential to dominate the entire space; that's because these are both low-level technologies that often back larger platforms such as Vuforia. Both of these platforms now support mobile pretrained networks for fast recognition on mobile devices. This may not seem like a big deal yet, but after we get into training our own models, you will see why.

Linear regression explained

Let's discuss the basic premise behind what Machine Learning is and what it attempts to accomplish. Take a look at the following chart that shows some fictional sales data for your next app:

Chart of fictional sales data

Now, just looking at the chart, you can see that as the x values increase (perhaps days on sale), it appears that our sales also increase: y value (sales). By just eyeing the chart, we ourselves can make predictions by following the trend of the points. Try it; how many sales are for an x value (bottom axis) of 25? Give it a guess, and write it down. With your guess secured, we will use a technique called linear regression to find a good answer.

Linear regression has been around for years and is considered as the base for many statistical data analysis methods. It is the basis for many other Machine Learning algorithms used in data science and predictive analysis today. This technique works by finding a solution (a line, curve, or whatever) that...