Azure Data Engineer Associate Certification Guide

Newton Alex

• Publisher: Packt Publishing
• Year: 2022
• eBook ISBN: 9781801812832
• Topic: Computer Science
• Subtopic: Data Modelling & Design
• Index: Computer Science

Become well-versed with data engineering concepts and exam objectives to achieve Azure Data Engineer Associate certification

Key Features

• Understand and apply data engineering concepts to real-world problems and prepare for the DP-203 certification exam
• Explore the various Azure services for building end-to-end data solutions
• Gain a solid understanding of building secure and sustainable data solutions using Azure services

Book Description

Azure is one of the leading cloud providers in the world, providing numerous services for data hosting and data processing. Most of the companies today are either cloud-native or are migrating to the cloud much faster than ever. This has led to an explosion of data engineering jobs, with aspiring and experienced data engineers trying to outshine each other.Gaining the DP-203: Azure Data Engineer Associate certification is a sure-fire way of showing future employers that you have what it takes to become an Azure Data Engineer. This book will help you prepare for the DP-203 examination in a structured way, covering all the topics specified in the syllabus with detailed explanations and exam tips. The book starts by covering the fundamentals of Azure, and then takes the example of a hypothetical company and walks you through the various stages of building data engineering solutions. Throughout the chapters, you'll learn about the various Azure components involved in building the data systems and will explore them using a wide range of real-world use cases. Finally, you'll work on sample questions and answers to familiarize yourself with the pattern of the exam.By the end of this Azure book, you'll have gained the confidence you need to pass the DP-203 exam with ease and land your dream job in data engineering.

What you will learn

• Gain intermediate-level knowledge of Azure the data infrastructure
• Design and implement data lake solutions with batch and stream pipelines
• Identify the partition strategies available in Azure storage technologies
• Implement different table geometries in Azure Synapse Analytics
• Use the transformations available in T-SQL, Spark, and Azure Data Factory
• Use Azure Databricks or Synapse Spark to process data using Notebooks
• Design security using RBAC, ACL, encryption, data masking, and more
• Monitor and optimize data pipelines with debugging tips

Who this book is for

This book is for data engineers who want to take the DP-203: Azure Data Engineer Associate exam and are looking to gain in-depth knowledge of the Azure cloud stack.The book will also help engineers and product managers who are new to Azure or interviewing with companies working on Azure technologies, to get hands-on experience of Azure data technologies. A basic understanding of cloud technologies, extract, transform, and load (ETL), and databases will help you get the most out of this book.

]]>

Part 1: Azure Basics

In this part, we focus on brushing up on the basics of Azure, including the IaaS, PaaS, and SaaS services that are available in Azure. We will cover topics including VMs, VNets, app services, Service Fabric, storage, managing services using the Azure portal, APIs, and command-line options.

This section comprises the following chapter:

• Chapter 1, Introducing Azure Basics

Chapter 1: Introducing Azure Basics

With all the initial formalities now behind us, let's start our journey in Azure. Our journey starts here: https://azure.microsoft.com.

Azure is one of the most important cloud platform providers on the market today. It provides several cloud, hybrid, and on-premises services such as VMs, networks, compute, databases, messaging, machine learning (ML), artificial intelligence, Internet of Things (IoT), and many more services while focusing on security and compliance. You could use these services to build anything from web pages and mobile apps, from data analytics solutions to IoT solutions and more.

In Azure, users have the flexibility to choose from completely hosted no-code solutions to completely build their solutions ground up using the basic building blocks like VMs and VNets, where the users have full control over each and every aspect of the system. And most of these technologies come prebaked with the cloud advantages, such as geo-replication, high availability, data redundancy, scalability, and elasticity.

Let's quickly review the basics of Azure. The following sections will focus on brushing up on the fundamentals of Azure. If you already have a working knowledge of Azure and know how to spin up resources in Azure, then you can safely skip this chapter and go directly to the next one.

In this first chapter, we'll provide an overview of Azure, including introducing some common Azure services. We'll get a good grounding in the basics, such as Accounts, virtual machines (VMs), storage, compute, and networking. We'll also walk through how to spin up services using both the Azure portal and the CLI.

In this chapter, we will cover the following topics:

• Introducing the Azure portal
• Exploring Azure accounts, subscriptions, and resource groups
• Introducing Azure services
• Exploring Azure VMs
• Exploring Azure storage
• Exploring Azure networking (VNet)
• Exploring Azure compute

Let's get started!

Technical requirements

To follow along with this chapter, you will need the following:

• An Azure account (free or paid)
• The Azure CLI installed on your workstation

Introducing the Azure portal

The Azure portal is the starting page for all Azure developers. You can think of it as an index page that contains links to all the services provided by Azure. The following screenshot shows what the Azure portal looks like:

Figure 1.1 – The Azure portal home page

You can browse through all the services available in Azure or quickly search for them using the search box. Once you click on a service, the corresponding service web page will appear (also known as blades in Azure). Azure maintains strong consistency in terms of blade design. All the service blades will look very similar. So, if you are familiar with one, you should be able to easily navigate the others. We will be exploring a few of the service blades in this chapter.

Exploring Azure accounts, subscriptions, and resource groups

You can explore Azure with or without an account. If you are just exploring Azure and are planning to run a few Sandbox experiments, you don't need to create an Azure account. But if you are planning on investing more time in Azure, then it is recommended to create an account. Azure provides USD 200 worth of free credits for the first 30 days for you to play around. This USD 200 should get you fairly good mileage for the practice exercises for this certification. You can enroll for a free account here: https://azure.microsoft.com/free.

Note

Azure requires a valid credit card number to create the account, but it doesn't charge the credit card for free accounts. Once the USD 200 credit is exhausted, it will notify you and then delete the resources.

Let us start with Azure accounts.

Azure account

An Azure account refers to the Azure Billing account. It is mapped to the email id that you used to sign up for Azure. An account can contain multiple subscriptions; each of these subscriptions can have multiple resource groups and the resource groups, in turn, can have multiple resources. The billing is done at the level of subscriptions. So, one account could have multiple invoices raised per subscription.

Next, let us look at Azure subscription.

Azure subscription

Every resource (VMs, VNets, databases, and so on) that you create in Azure is tied to a subscription. A subscription is a container for all the resources that are created for applications and solutions under that subscription. A subscription contains the details of all the VMs, networks, storage, and other services that were used during that month that will be used for billing purposes. Azure creates a default subscription when you create an account. But you could choose to have multiple subscriptions based on your teams (dev, test, sales, marketing, finance, and so on), regions (North America, EMEA, Asia Pacific, and so on), or other logical divisions that you feel are appropriate for your use case.

Next, let us look at Resource groups.

Resource groups

Resource groups are logical groups of resources belonging to an application or a team. You can think of them as tags associated with the resources so that you can easily query, monitor, and manage the collection of resources as one. For example, you could create a resource group called Sandbox for the Azure practice sessions. At the end of the day, you can delete all the resources that were created under that resource group in one go, instead of going through every resource and deleting them. You can have multiple resourc...

Table of contents

• Azure Data Engineer Associate Certification Guide
• Contributors
• Preface
• Part 1: Azure Basics
• Chapter 1: Introducing Azure Basics
• Part 2: Data Storage
• Chapter 2: Designing a Data Storage Structure
• Chapter 3: Designing a Partition Strategy
• Chapter 4: Designing the Serving Layer
• Chapter 5: Implementing Physical Data Storage Structures
• Chapter 6: Implementing Logical Data Structures
• Chapter 7: Implementing the Serving Layer
• Part 3: Design and Develop Data Processing (25-30%)
• Chapter 8: Ingesting and Transforming Data
• Chapter 9: Designing and Developing a Batch Processing Solution
• Chapter 10: Designing and Developing a Stream Processing Solution
• Chapter 11: Managing Batches and Pipelines
• Part 4: Design and Implement Data Security (10-15%)
• Chapter 12: Designing Security for Data Policies and Standards
• Part 5: Monitor and Optimize Data Storage and Data Processing (10-15%)
• Chapter 13: Monitoring Data Storage and Data Processing
• Chapter 14: Optimizing and Troubleshooting Data Storage and Data Processing
• Part 6: Practice Exercises
• Chapter 15: Sample Questions with Solutions
• Other Books You May Enjoy