Learn Unity ML-Agents – Fundamentals of Unity Machine Learning

Incorporate new powerful ML algorithms such as Deep Reinforcement Learning for games

Micheal Lanham

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781789131864
• Edition: 1
• Topic: Computer Science
• Subtopic: Programming in Python
• Index: Computer Science

Transform games into environments using machine learning and Deep learning with Tensorflow, Keras, and Unity

Key Features

• Learn how to apply core machine learning concepts to your games with Unity
• Learn the Fundamentals of Reinforcement Learning and Q-Learning and apply them to your games
• Learn How to build multiple asynchronous agents and run them in a training scenario

Book Description

Unity Machine Learning agents allow researchers and developers to create games and simulations using the Unity Editor, which serves as an environment where intelligent agents can be trained with machine learning methods through a simple-to-use Python API.

This book takes you from the basics of Reinforcement and Q Learning to building Deep Recurrent Q-Network agents that cooperate or compete in a multi-agent ecosystem. You will start with the basics of Reinforcement Learning and how to apply it to problems. Then you will learn how to build self-learning advanced neural networks with Python and Keras/TensorFlow. From there you move o n to more advanced training scenarios where you will learn further innovative ways to train your network with A3C, imitation, and curriculum learning models. By the end of the book, you will have learned how to build more complex environments by building a cooperative and competitive multi-agent ecosystem.

What you will learn

• Develop Reinforcement and Deep Reinforcement Learning for games.
• Understand complex and advanced concepts of reinforcement learning and neural networks
• Explore various training strategies for cooperative and competitive agent development
• Adapt the basic script components of Academy, Agent, and Brain to be used with Q Learning.
• Enhance the Q Learning model with improved training strategies such as Greedy-Epsilon exploration
• Implement a simple NN with Keras and use it as an external brain in Unity
• Understand how to add LTSM blocks to an existing DQN
• Build multiple asynchronous agents and run them in a training scenario

Who this book is for

This book is intended for developers with an interest in using Machine learning algorithms to develop better games and simulations with Unity.

The reader will be required to have a working knowledge of C# and a basic understanding of Python.

Terrarium Revisited – A Multi-Agent Ecosystem

In 2002, Microsoft developed a fun little developer game called Terrarium that demonstrated the code portability of the .NET Framework. Back then, C# and .NET were just newcomers and Microsoft had a tough sell; after all, it was dethroning VBA and Visual Basic as well as trying to steal away Java developers. The game allowed developers to build tiny programmable creatures that would be born, eat, sleep, reproduce, and die in a terrarium. It also featured the ability of allowing tiny programs to infect other connected terrariums. In fact, Microsoft has a contest that pitted these tiny creature programs against each other and the winning developer was the one that had the most successful creature. While we won't go so far as to build a connected terrarium infrastructure, we will do our best to replicate a number of the cooler features that made the original game so entertaining (as a developer, anyway).

The focus of this chapter will be to develop a similar terrarium game which allows our agents to live and die, but with a few twists. While neural nets and other ML technologies were available at the time, virtually all the terrarium creatures were based on A* and heuristic methods that were created by developers. Now, we have the ability to let the little terrarium creatures program themselves and develop their own winning behaviors, possibly in ways we have never even thought about. This is going to be an exciting chapter and here is what we will cover:

• What Terrarium was/is
• Building the Agent ecosystem
• Basic Terrarium – Plants and Herbivores
• Carnivore: the hunter
• Next steps
• Exercises

In this chapter, we have a lot of ground to cover and we will need to go over some advanced Unity concepts quickly. If you find yourself in trouble, download the book's source code and look at the completed example.

If the concept of growing your own little programmable creatures in a terrarium doesn't excite you, then you probably picked this book up by accident. Please give it back to its rightful owner, thanks.

What was/is Terrarium?

Terrarium was a concept that was probably ahead of its time. While the game did allow developers to transfer some form of state between parents and offspring, it was a feature that was never used in successful agents. Instead, developers just optimized their code for a particular set of fixed strategies, which in many cases turned out to be a lot of code. The following is an example screenshot of a Terrarium client running in ecosystem mode:

Original Microsoft Terrarium

Now, the bottom line is that we are not going to be able to replicate the entire Terrarium connected ecosystem concept in one chapter. The elegance that was the Terrarium network ecosystem infrastructure could possibly take several chapters to explain. Instead, our goal is going to build a multi-agent multi-brain ecosystem that is designed to teach you more about building an ML-Agents environment and Deep Reinforcement Learning. Perhaps someone like Microsoft or Unity will take the connected ecosystem concept further – who knows.

The Terrarium game rules were fairly simple and one of the basic rules was that your code had to run in x amount of time without failure or the creature died. There were other aspects that allowed you to apply points into a creature's abilities like size, speed, toughness, and so on that we will focus on. Developers had 100 points that they could apply to various categories for creatures of type carnivore and herbivore, as shown in the following sample tables:

Attributes	Herbivore	Carnivore	Description
Max Energy	20	10	Determines the amount of energy the creature can store for actions. Carnivores consume energy by eating other creatures. Herbivores eat plants.
Eating Speed	2	6	Determines how fast the creature can consume energy from another creature.
Attack Damage	0	14	Amount of damage another creature causes in an attack.
Defend Damage	10	10	Determines how effective a creature defends from an attack. A creature with a higher defense will inflict damage back on the attacker.
Max Speed	8	10	Determines how fast a creature moves.
Camouflage	25	5	How difficult it is to see the creature.
Eyesight	5	20	How far away a creature can detect other creatures.
Mature Size	30	20	We are adding Mature Size to the original parameters in order to hopefully make the game more balanced.
Growth Rate	5	5	This is another parameter that has been added to manage creature growth a little cleaner. The original game handled creature growth a bit like a black box.

On top of this, all creatures have a Mature Size attribute that can vary from 24 to 48, which corresponded to the icon's image size in the original game. We will map all of these attributes to various game mechanics when we build the example, but for us to perform reinforcement learning, we are going to need to apply some form of reward structure as well.

Previously, all of our rewards have been goal orientated; get to the goal and get a reward. With Terrarium however, the goal was and still is to make our agents live as long as possible and reproduce. Therefore, we want to introduce a reward system that also mimics this. We will do this by giving rewards based on the following events:

Event	Reward	Description
Die of old age	+1	Our goal will be for the creature agents to survive as long as possible.
Reproduce	+1	We want to encourage reproduction; more agents means more learning and food for others.
Eat	+energy eaten/100	We give this micro reward in order to encourage creatures to eat. Without it, they may realize they don't need to do this and just continually die of starvation. Besides, everyone enjoys e...