Building Big Data Applications

Krish Krishnan

• Publisher: Academic Press
• Year: 2019
• Print ISBN: 9780128157466
• eBook ISBN: 9780128158043
• Topic: Technology & Engineering
• Subtopic: Business Intelligence
• Index: Technology & Engineering

Building Big Data Applications helps data managers and their organizations make the most of unstructured data with an existing data warehouse. It provides readers with what they need to know to make sense of how Big Data fits into the world of Data Warehousing. Readers will learn about infrastructure options and integration and come away with a solid understanding on how to leverage various architectures for integration. The book includes a wide range of use cases that will help data managers visualize reference architectures in the context of specific industries (healthcare, big oil, transportation, software, etc.).- Explores various ways to leverage Big Data by effectively integrating it into the data warehouse- Includes real-world case studies which clearly demonstrate Big Data technologies- Provides insights on how to optimize current data warehouse infrastructure and integrate newer infrastructure matching data processing workloads and requirements

1

Big Data introduction

Abstract

This chapter presents an introduction to Big Data. The world we live in today is flooded with data. It delivers business value and ranges from personal care to beauty, healthily eating, clothing, perfumes, watches, jewelry, medicine, travel, tours, and investments. Big Data Applications are the answer to leveraging the analytics from complex events and getting the articulate insights for the enterprise. We should define a metadata-driven architecture to integrate the data for creating analytics. More opportunities exist in terms of space exploration, smart cars and trucks, and new forays into energy research as well as the smart wearable devices and devices for pet monitoring, remote communications, healthcare monitoring, sports training, and many other innovations.

Keywords

Analytics; Big Data; Hadoop technology; Healthcare monitoring; Remote communications; SAP

This chapter will be a brief introduction to Big Data, providing readers the history, where are we today, and the future of data. The reader will get a refresher view of the topic.

The world we live in today is flooded with data all around us, produced at rates that we have not experienced, and analyzed for usage at rates that we have heard as requirements before and now can fulfill the request. What is the phenomenon called as “Big Data” and how has it transformed our lives today? Let us take a look back at history, in 2001 when Doug Laney was working with Meta Group, he forecasted a trend that will create a new wave of innovation and articulated that the trend will be driven by the three V's namely volume, velocity, and variety of data. In the continuum in 2009, he wrote the first premise on how “Big Data” as the term was coined by him will impact the lives of all consumers using it. A more radical rush was seen in the industry with the embracement of Hadoop technology and followed by NoSQL technologies of different varieties, ultimately driving the evolution of new data visualization, analytics, storyboarding,and storytelling.

In a lighter vein, SAP published a cartoon which read the four words that Big Data brings —“Make Me More Money”

This is the confusion we need to steer clear of and be ready to understand how to monetize from Big Data.

First to understand how to build applications with Big Data, we need to look at Big Data from both the technology and data perspectives.

Big Data delivers business value

The e-Commerce market has shaped businesses around the world into a competitive platform where we can sell and buy what we need based on costs, quality, and preference. The spread of services ranges from personal care, beauty, healthily eating, clothing, perfumes, watches, jewelry, medicine, travel, tours, investments, and the list goes on. All of this activity has resulted in data of various formats, sizes, languages, symbols, currencies, volumes, and additional metadata which we collectivity today call as “Big Data”. The phenomenon has driven unprecedented value to business and can deliver insights like never before.

The business value did not and does not stop here; we are seeing the use of the same techniques of Big Data processing across insurance, healthcare, research, physics, cancer treatment, fraud analytics, manufacturing, retail, banking, mortgage, and more. The biggest question is how to realize the value repeatedly? What formula will bring success and value, how to monetize from the effort?

Take a step back for a moment and assess the same question with investments that has been made into a Salesforce or Unica or Endeca implementation and the business value that you can drive from the same. Chances are you will not have an accurate picture of the amount of return on investmentor the percentage of impact in terms of increased revenue or decreased spendor process optimization percentages from any such prior experiences. Not that your teams did not measure the impact, but they are unsure of expressing the actual benefit into quantified metrics. But in the case of a Big Data implementation, there are techniques to establish a quantified measurement strategy and associate the overall program with such cost benefits and process optimizations.

The interesting question to ask is what are organizations doing with Big Data? Are they collecting it, studying it, and working with it for advanced analytics? How exactly does the puzzle called Big Data fit into an organization's strategy and how does it enhance corporate decision-making?

To understand this picture better there are some key questions to think about and these are a few you can add more to this list:

1. • How many days does it take on an average to get answers to the question “why”?
2. • How many cycles of research does the organization do for understanding the market, competition, sales, employee performance, and customer satisfaction?
3. • Can your organization provide an executive dashboard along the ZachmanFramework model to provide insights and business answers on who, what, where, when, and how?
4. • Can we have a low code application that will be orchestrated with a workflow and can provide metrics and indicators on key processes?
5. • Do you have volumes of data but have no idea how to use it or do not collect it at all?
6. • Do you have issues with historical analysis?
7. • Do you experience issues with how to replay events? Simple or complex events?

The focus of answering these questions through the eyes of data is very essential and there is an abundance of data that any organization has today and there is a lot of hidden data or information in these nuggets that have to be harvested. Consider the following data:

1. • Traditional business systems—ERP, SCM, CRM, SFA
2. • Content management platforms
3. • Portals
4. • Websites
5. • Third-party agency data
6. • Data collected from social media
7. • Statistical data
8. • Research and competitive analysis data
9. • Point of sale data—retail or web channel
10. • Legal contracts
11. • Emails

If you observe a pattern here there is data about customers, products, services, sentiments, competition, compliance, and much more available. The question is does the organization leverage all the data that is listed here? And more important is the question, can you access all this data at relative ease and implement decisions? This is where the platforms and analytics of Big Data come into the picture within the enterprise. From the data nuggets that we have described 50% of them or more are internal systems and data producers that have been used for gathering data but not harnessing analytical value (the data here is structured, semistructured, and unstructured), the other 50% or less is the new data that is called Big Data (web data, machine data, and sensor data).

Big Data Applications are the answer to leveraging the analytics from complex events and getting the articulate insights for the enterprise. Consider the following example:

1. • Call center optimization—The worst fear of a customer is to deal with the call center. The fundamental frustration for the customer is the need to explain all the details about their transactions with the company they are calling, the current situation, and what they are expecting for a resolution, not once but many times (in most cases) to many people and maybe in more than one conversation. All of this frustration can be vented on their Facebook page or Twitter or a social media blog, causing multiple issues
2. 1. • They will have an influence in their personal network that will cause potential attrition of prospects and customers
   2. • Their frustration maybe shared by many others and eventually result in class action lawsuits
   3. • Their frustration will provide an opportunity for the competition to pursue and sway customers and prospects
   4. • All of these actions lead to one factor called as “revenue loss.”If this company continues to persist with poor quality of service, eventually the losses will be large and even leading to closure of business and loss of brand reputation. It is in situations like this where you can find a lot of knowledge in connecting the dots with data and create a powerful set of analytics to drive business transformation. Business transformation does not mean you need to change your operating model but rather it provides opportunities to create new service models created on data driven decisions and analytics.

The company that we are discussing here, let us assume,decides that the current solution needs an overhaul and the customer needs to be provided the best quality of service, it will need to have the following types of data ready for analysis and usage:

1. • Customer profile, lifetime value, transactional history, segmentation models, social profiles (if provided)
2. • Customer sentiments, survey feedback, call center interactions
3. • Product analytics
4. • Competitive research
5. • Contracts and agreements—customer specific

We should define a metadata-driven architecture to integrate the data for creating these analytics. There is a nuance of selecting the right technology and architecture for the physical deployment. A few days later the customer calls for support, the call center agent is now having a mash-up showing different types of analytics presented to them. The agent is able to ask the customer-guided questions on the current call and apprise them of the solutions and timelines, rather than ask for information; they are providing a knowledge service. In this situation the customer feels more privileged and even if there are issues with the service or product, the customer will not likely attrite. Furthermore, the same customer now can share positive feedback and report their satisfaction, thus creating a potential opportunity for more revenue. The agent feels more empowered and can start having conversat...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• Dedication
• Preface
• 1. Big Data introduction
• 2. Infrastructure and technology
• 3. Building big data applications
• 4. Scientific research applications and usage
• 5. Pharmacy industry applications and usage
• 6. Visualization, storyboarding and applications
• 7. Banking industry applications and usage
• 8. Travel and tourism industry applications and usage
• 9. Governance
• 10. Building the big data application
• 11. Data discovery and connectivity
• Use cases from industry vendors
• Index