Machine Learning Engineering with MLflow

Natu Lauchande

• Publisher: Packt Publishing
• Year: 2021
• ISBN:
  9781800561694
• Edition: 1
• Topic: Ciencia de la computación
• Subtopic: Modelado y diseño de datos

Get up and running, and productive in no time with MLflow using the most effective machine learning engineering approachKey Features• Explore machine learning workflows for stating ML problems in a concise and clear manner using MLflow• Use MLflow to iteratively develop a ML model and manage it • Discover and work with the features available in MLflow to seamlessly take a model from the development phase to a production environmentBook DescriptionMLflow is a platform for the machine learning life cycle that enables structured development and iteration of machine learning models and a seamless transition into scalable production environments.This book will take you through the different features of MLflow and how you can implement them in your ML project. You will begin by framing an ML problem and then transform your solution with MLflow, adding a workbench environment, training infrastructure, data management, model management, experimentation, and state-of-the-art ML deployment techniques on the cloud and premises. The book also explores techniques to scale up your workflow as well as performance monitoring techniques. As you progress, you'll discover how to create an operational dashboard to manage machine learning systems. Later, you will learn how you can use MLflow in the AutoML, anomaly detection, and deep learning context with the help of use cases. In addition to this, you will understand how to use machine learning platforms for local development as well as for cloud and managed environments. This book will also show you how to use MLflow in non-Python-based languages such as R and Java, along with covering approaches to extend MLflow with Plugins.By the end of this machine learning book, you will be able to produce and deploy reliable machine learning algorithms using MLflow in multiple environments.What you will learn• Develop your machine learning project locally with MLflow's different features• Set up a centralized MLflow tracking server to manage multiple MLflow experiments• Create a model life cycle with MLflow by creating custom models• Use feature streams to log model results with MLflow• Develop the complete training pipeline infrastructure using MLflow features• Set up an inference-based API pipeline and batch pipeline in MLflow• Scale large volumes of data by integrating MLflow with high-performance big data librariesWho this book is forThis book is for data scientists, machine learning engineers, and data engineers who want to gain hands-on machine learning engineering experience and learn how they can manage an end-to-end machine learning life cycle with the help of MLflow. Intermediate-level knowledge of the Python programming language is expected.

Section 1: Problem Framing and Introductions

This section will introduce you to a framework for stating machine learning problems in a concise and clear manner using MLflow.

The following chapters are covered in this section:

• Chapter 1, Introducing MLflow
• Chapter 2, Your Machine Learning Project

Chapter 1: Introducing MLflow

MLflow is an open source platform for the machine learning (ML) life cycle, with a focus on reproducibility, training, and deployment. It is based on an open interface design and is able to work with any language or platform, with clients in Python and Java, and is accessible through a REST API. Scalability is also an important benefit that an ML developer can leverage with MLflow.

In this chapter of the book, we will take a look at how MLflow works, with the help of examples and sample code. This will build the necessary foundation for the rest of the book in order to use the concept to engineer an end-to-end ML project.

Specifically, we will look at the following sections in this chapter:

• What is MLflow?
• Getting started with MLflow
• Exploring MLflow modules

Technical requirements

For this chapter, you will need the following prerequisites:

• The latest version of Docker installed in your machine. In case you don't have the latest version, please follow the instructions at the following URL: https://docs.docker.com/get-docker/.
• Access to a bash terminal (Linux or Windows).
• Access to a browser.
• Python 3.5+ installed.
• PIP installed.

What is MLflow?

Implementing a product based on ML can be a laborious task. There is a general need to reduce the friction between different steps of the ML development life cycle, and between teams of data scientists and engineers that are involved in the process.

ML practitioners, such as data scientists and ML engineers, operate with different systems, standards, and tools. While data scientists spend most of their time developing models in tools such as Jupyter Notebooks, when running in production, the model is deployed in the context of a software application with an environment that is more demanding in terms of scale and reliability.

A common occurrence in ML projects is to have the models reimplemented by an engineering team, creating a custom-made system to serve the specific model. A set of challenges are common with teams that follow bespoke approaches regarding model development:

• ML projects that run over budget due to the need to create bespoke software infrastructure to develop and serve models
• Translation errors when reimplementing the models produced by data scientists
• Scalability issues when serving predictions
• Friction in terms of reproducing training processes between data scientists due to a lack of standard environments

Companies leveraging ML tend to create their own (often extremely laborious) internal systems in order to ensure a smooth and structured process of ML development. Widely documented ML platforms include systems such as Michelangelo and FBLearner, from Uber and Facebook, respectively.

It is in the context of the increasing adoption of ML that MLflow was initially created at Databricks and open sourced as a platform, to aid in the implementation of ML systems.

MLflow enables an everyday practitioner in one platform to manage the ML life cycle, from iteration on model development up to deployment in a reliable and scalable environment that is compatible with modern software system requirements.

Getting started with MLflow

Next, we will install MLflow on your machine and prepare it for use in this chapter. You will have two options when it comes to installing MLflow. The first option is through a Docker container-based recipe provided in the repository of the book: https://github.com/PacktPublishing/Machine-Learning-Engineering-with-Mlflow.git.

To install it, follow these instructions:

1. Use the following commands to install the software:
   $ git clone https://github.com/PacktPublishing/Machine-Learning-Engineering-with-Mlflow.git
   $ cd Machine-Learning-Engineering-with-Mlflow
   $ cd Chapter01
2. The Docker image is very simple at this stage: it simply contains MLflow and sklearn, the main tools to be used in this chapter of the book. For illustrative purposes, you can look at the content of the Dockerfile:
   FROM jupyter/scipy-notebook
   RUN pip install mlflow
   RUN pip install sklearn
3. To build the image, you should now run the following command:
   docker build -t chapter_1_homlflow
4. Right after building the image, you can run the ./run.sh command:
   ./run.sh
   Important note
   It is important to ensure that you have the latest version of Docker installed on your machine.
5. Open your browser to http://localhost:888 and you should be able to navigate to the Chapter01 folder.

In the following section, we will be developing our first model with MLflow in the Jupyter environment created in the previous set of steps.

Developing your first model with M...