Build Real Time Data Analytics on Google Cloud Platform

Build Real Time Data Analytics on Google Cloud Platform

Murari Ramuka

• Publisher: BPB Publications
• Year: 2019
• eBook ISBN: 9789389423631
• Topic: Computer Science
• Subtopic: Cloud Computing
• Index: Computer Science

Step-by-step guide to different data movement and processing techniques, using Google Cloud Platform Services Key Features

• Learn the basic concept of Cloud Computing along with different Cloud service provides with their supported Models (IaaS/PaaS/SaaS)
• Learn the basics of Compute Engine, App Engine, Container Engine, Project and Billing setup in the Google Cloud Platform
• Learn how and when to use Cloud DataFlow, Cloud DataProc and Cloud DataPrep
• Build real-time data pipeline to support real-time analytics using Pub/Sub messaging service
• Setting up a fully managed GCP Big Data Cluster using Cloud DataProc for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient manner
• Learn how to use Cloud Data Studio for visualizing the data on top of Big Query
• Implement and understand real-world business scenarios for Machine Learning, Data Pipeline Engineering
• 
  Description
  Modern businesses are awash with data, making data-driven decision-making tasks increasingly complex. As a result, relevant technical expertise and analytical skills are required to do such tasks. This book aims to equip you with enough knowledge of Cloud Computing in conjunction with Google Cloud Data platform to succeed in the role of a Cloud data expert.
  The current market is trending towards the latest cloud technologies, which is the need of the hour. Google being the pioneer, is dominating this space with the right set of cloud services being offered as part of GCP (Google Cloud Platform). At this juncture, this book will be very vital and will be cover all the services that are being offered by GCP, putting emphasis on Data services. What will you learn
  By the end of the book, you will have come across different data services and platforms offered by Google Cloud, and how those services/features can be enabled to serve business needs. You will also see a few case studies to put your knowledge to practice and solve business problems such as building a real-time streaming pipeline engine, Scalable Datawarehouse on Cloud, fully managed Hadoop cluster on Cloud and enabling TensorFlow/Machine Learning API's to support real-life business problems. Remember to practice additional examples to master these techniques. Who this book is for
  This book is for professionals as well as graduates who want to build a career in Google Cloud data analytics technologies. One-stop shop for those who wish to get an initial to advance understanding of the GCP data platform. The target audience will be data engineers/professionals who are new, as well as those who are acquainted with the tools and techniques related to cloud and data space.
  ? Individuals who have basic data understanding (i.e. Data and cloud) and have done some work in the field of data analytics, can refer/use this book to master their knowledge/understanding.
  ? The highlight of this book is that it will start with the basic cloud computing fundamentals and will move on to cover the advance concepts on GCP cloud data analytics and hence can be referred across multiple different levels of audiences. Table of Contents
  1. GCP Overview and Architecture
  2. Data Storage in GCP
  3. Data Processing in GCP with Pub/Sub and Dataflow
  4. Data Processing in GCP with DataPrep and Dataflow
  5. Big Query and Data Studio
  6. Machine Learning with GCP
  7. Sample Use cases and Examples
  

CHAPTER 1

GCP Overview and Architecture

Introduction

Cloud computing is a buzz word in the market these days. Every organization wants to go for cloud first strategy where they can get the power of cloud computing. In this chapter, we will start with basic cloud computing concepts and discuss on Google Cloud Platforms (GCP), including their different cloud service offerings.

Structure

• Cloud computing
• Cloud models
• Major Cloud Vendors
• Google Cloud Platform
• Region and zone
• Important service in Google Cloud Platform

Objectives

The objective of this chapter is to explain the different cloud computing models and major vendors. You will get well versed with the GCP architecture, region, zone, billing, roles, and features along with different types of GCP services and can select appropriate services based on specific requirements.

In the earlier days, organizations were directly responsible for managing their own infrastructure. These infrastructures include servers, storages, and computing powers. Maintenance of these infrastructures was a tedious task and incurred enormous expenses to the companies. These drawbacks and unexpected initial cost of setting up an infrastructure triggered the need for cloud computing models.

Cloud computing helps organizations cater to on-demand availability of all the computer resources (that is, data storage, computing power, networks, and applications) without taking the ownership of their management. It is widely distributed over multiple network which supports huge data storage and computing power. In today’s time, there is a central server for most of the large clouds, which have functions distributed over various locations. Basically, it is using someone else’s server to host, run, and process any application along with storing the data. Hence, cloud computing enables enterprises to avoid or minimize different IT services costs upfront (that is, infrastructure, application deployment, and more). Not only just cost, but it helps organizations to set up and run their applications faster with improved insight and maintenance. The IT team can well handle the fluctuation of application demands during peak and off-peak hours via cloud, which is one of the very important features of cloud. Autoscaling, which is one of important features, helps in this type of scenario. Pay-as-you-go helps enterprises to select a proper costing model to support their infra and other services requirements. The following features led to foundation and growth of cloud computing:

• Availability of high-capacity networks
• Low-cost computers and storage devices
• Common adoption of hardware virtualization
• Service-oriented architecture
• Autonomic and utility computing
• Pay-as-you-go
• Autoscaling

Cloud computing history

Cloud computing has been in existence from early 2000. Amazon created subsidiary organization called Amazon Web Services in August 2006 and introduced its main service which is Elastic Compute Cloud (EC2). In April 2008, Google also came in to cloud space and released Google App Engine with their beta release. In February 2010, Microsoft released Microsoft Azure, which was announced in October 2008. On March 1, 2011, IBM followed cloud race and announced the IBM SmartCloud framework to support Smarter Planet. Google Compute Engine, which is one of the services under GCP, was released in preview in May 2012, before being rolled out into general availability in December 2013.

On-premise versus cloud computing

It has been always a debate on pros and cons of on-premise and cloud infrastructure. Both have some advantages and disadvantages, which are listed as follows:

Factors	On-premise	Cloud computing
Deployment	Deployment of the resources is within the infrastructure. The organization will be responsible for maintaining and handling the deployment related process. Since the application is hosted within, the access is limited to the organization only.	In cloud computing, resources are deployed at the service provider's end and accessed by the public. In private cloud, resources and application are deployed according to the customer’s need and can be accessed by them only.
Cost	The initial cost includes servers, hardware, storage devices, software, power consumption and space where architecture is built. Hence, the initial cost is high.	In cloud computing, the users only need to pay for the resources they use. There are no maintenance charge, no upfront charge, and no upkeep costs associated.
Security	Organizations having sensitive data, for example, agencies must use a certain level of security. The security is taken care by either a third party or by a group of staff using an external tool.	The secure environment is provided by the cloud service providers. There is a broad set of policies and technologies provided by the CSPs. These take care of the security of enterprise data.
Flexibility	When any infrastructure upgrade/changes need to be applied, the cost incurred will be by the organization.	Enterprise can quickly upgrade their infrastructure to their requirements without having to make large investments in costly hardware every time.
Maintenance	The user/enterprises are responsible for maintaining the server hardware and software, the data backups, storage devices, and disaster recovery.	Cloud computing provides greater flexibility as the user/organization only pay for what they use and can easily scale to meet the peak demand.

Considering the preceding highlighted differences, anyone can easily differentiate the advantages of using cloud over on-prem. As per National Institute of Standards and Technology (NIST), the definition of cloud computing identifies five essential characteristics, which are as follows:

• On-demand self-service: A consumer can unilaterally provision computing capabilities such as server time and network storage, as needed automatically without needing human interaction with each service provider.
• Broad network access: Capabilities are available over the network and accessed via standard mechanisms that promote use of heterogeneous thin or thick client platforms (for example, mobile phones, tablets, laptops, and workstations).
• Resource pooling: The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify a location at a higher level of abstraction (for example, country, state, or datacenter). Examples of resources include storage, processing, memory, and network bandwidth.
• Rapid elasticity: Capabilities can be elastically provisioned and released, and even automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time.
• Measured service: Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (for example, storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

Benefits of cloud computing

Globally cloud computing has created a deep impact on innovation, and therefore, the political economy of any business and country. It permits business and organization with innovative concept to add an additional chance not solely to enhance flexibility, scale back prices, and specialize in core competencies, however, conjointly to completely rework no matter how they operate. For instance, by re-designing internal system workflows or client interactions that permits digital experiences from mobile devices to any or all the thanks to the cloud information centers.

Specifically, the business benefits of cloud computing includes:

• Various cloud services that incorporates storage, compute, network, and more is purchased and consumed on a pay-as-you-go basis and redoubled or diminished as required for optimum utilization.
• Cloud computing helps to convert capital expenses into operation expenses and therefore up the potency.
• Since there's no software system is put in, configured, or upgraded on persona...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Disclaimer
• Dedication
• About the Author
• About the Reviewer
• Acknowledgement
• Preface
• Errata
• Table of Contents
• 1. GCP Overview and Architecture
• 2. Google Cloud Platform Storage
• 3. Data Processing and Message with Dataflow and Pub/Sub
• 4. Data Processing with Dataproc and Dataprep
• 5. BigQuery and Data Studio
• 6. Machine Learning with GCP
• 7. Sample Use Cases and Example