Artificial Intelligence for Big Data

Complete guide to automating Big Data solutions using Artificial Intelligence techniques

Anand Deshpande, Manish Kumar

• Publisher: Packt Publishing
• Year: 2018
• Print ISBN: 9781788472173
• eBook ISBN: 9781788476010
• Edition: 1
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Build next-generation Artificial Intelligence systems with Java

Key Features

• Implement AI techniques to build smart applications using Deeplearning4j
• Perform big data analytics to derive quality insights using Spark MLlib
• Create self-learning systems using neural networks, NLP, and reinforcement learning

Book Description

In this age of big data, companies have larger amount of consumer data than ever before, far more than what the current technologies can ever hope to keep up with. However, Artificial Intelligence closes the gap by moving past human limitations in order to analyze data.

With the help of Artificial Intelligence for big data, you will learn to use Machine Learning algorithms such as k-means, SVM, RBF, and regression to perform advanced data analysis. You will understand the current status of Machine and Deep Learning techniques to work on Genetic and Neuro-Fuzzy algorithms. In addition, you will explore how to develop Artificial Intelligence algorithms to learn from data, why they are necessary, and how they can help solve real-world problems.

By the end of this book, you'll have learned how to implement various Artificial Intelligence algorithms for your big data systems and integrate them into your product offerings such as reinforcement learning, natural language processing, image recognition, genetic algorithms, and fuzzy logic systems.

What you will learn

• Manage Artificial Intelligence techniques for big data with Java
• Build smart systems to analyze data for enhanced customer experience
• Learn to use Artificial Intelligence frameworks for big data
• Understand complex problems with algorithms and Neuro-Fuzzy systems
• Design stratagems to leverage data using Machine Learning process
• Apply Deep Learning techniques to prepare data for modeling
• Construct models that learn from data using open source tools
• Analyze big data problems using scalable Machine Learning algorithms

Who this book is for

This book is for you if you are a data scientist, big data professional, or novice who has basic knowledge of big data and wish to get proficiency in Artificial Intelligence techniques for big data. Some competence in mathematics is an added advantage in the field of elementary linear algebra and calculus.

Deep Big Data Analytics

In the previous chapter, we established the fundamental theory of artificial neural networks (ANNs) and how they emulate human brain structure for generating output based on a set of inputs with the help of interconnected nodes. The nodes are arranged in three types of layers: input, hidden, and output. We understood the basic and mathematical concepts of how the input signal is carried through to the output layer and the iterative approach that ANNs take for training weights on neuron connections. Simple neural networks with one or two hidden layers can solve very rudimentary problems. However, in order to meaningfully utilize ANNs for real-world problems, which involve hundreds or thousands of input variables, involve more complex models, and require the models to store more information, we need more complex structures that are realized with large numbers of hidden layers. These types of networks are called Deep Neural Networks and utilizing these Deep Neural Networks for modeling the real data is termed deep learning. With the addition of nodes and their interconnections, the Deep Neural Networks can model unstructured input, such as audio, video, and images.

In this chapter, we will explore how deep learning can be utilized for addressing some important problems in big data analytics, including extracting complex patterns from massive volumes of data, semantic indexing, data tagging, fast information retrieval, and simplifying discriminative tasks such as classification. We are going to cover:

• The building blocks of deep learning:
  • Gradient descent
  • Backpropagation
  • Non-linearities
  • Dropout
• Specialized neural net architectures for structured data
• Building data preparation pipelines
• Hyperparameter tuning
• Leveraging distributed computing for deep learning

The proposed examples will be implemented using the Deeplearning4j (DL4J) Java framework.

Deep learning basics and the building blocks

In the previous chapters, we established the fact that the machine learning algorithms generalize the input data into a hypothesis that fits the data so that the output, based on the new values, can be predicted accurately by the model. The accuracy of the model is a function of the amount of the input data along with variation in the values of the independent variables. The more data and variety, the more computation power we require to generate and execute the models. The distributed computing frameworks (Hadoop, Spark, and so on) work very well with the large volumes of data with variety. The same principles apply to ANNs.

The more input data we have along with variations, the more accurate the models can be generated, which requires more storage and computation power. Since the computation power and storage is available with the development of the big data analytics platforms (in-premise as well as on the cloud), it is possible to experiment with large neural networks with hundreds or thousands of nodes in the input layer, and hundreds or thousands of hidden layers. These types of ANNs are called Deep Neural Networks.

While these models are computationally heavy, they produce accurate results and get better with more data, unlike the traditional algorithms that plateau in terms of performance at some point. After the plateau point, even after adding more data, the model accuracy for traditional mathematical models does not increase by a great margin. The Deep Neural Networks perform better in terms of accuracy and reliability with increasing amount of data. The use of these multi-layered neural networks for hypothesis generation is generally termed deep learning. The difference between a Simple Neural Network and a Deep Neural Network can be depicted as follows:

Simple ANN versus Deep Neural Network

For supervised learning problems, the Deep Neural Networks have proven to provide encouraging results, especially when it comes to mapping some of the functions with high complexity levels. With sufficiently large datasets with labeled training examples, the Deep Neural Networks are able to train the connection weights so that there is no loss of intelligence and the model accurately represents the historical facts based on data, and at the same time has a level of generalization that suits most of the mission critical applications. Remember, the generic and common objective of all the learning methods is to minimize the cost function. The cost function value is inversely proportional to the model's accuracy.

Let us mathematically define the cost function for a Deep Neural Network. This is also termed the mean squared error function. This function will always be positive since it takes the square of the difference:

w: collection of all the weights in the network b: all the biases n: training data size (number of samples) a: vector of outputs from the network corresponding to x as input value

Let's look at some of the methods of Deep Neural Networks learning.

Gradient-based learning

In the previous chapter, we primarily discussed the single hidden layer perceptron model or the simple neural networks, in that chapter we also introduced the concept of gradient descent. Gradient descent, as applicable to the Deep Neural Network, essentially means we define the weights and biases for the neuron connections so as to reduce the value of the cost function. The network is initialized to a random state (random weights and bias values...

Table of contents

• Title Page
• Copyright and Credits
• Packt Upsell
• Contributors
• Preface
• Big Data and Artificial Intelligence Systems
• Ontology for Big Data
• Learning from Big Data
• Neural Network for Big Data
• Deep Big Data Analytics
• Natural Language Processing
• Fuzzy Systems
• Genetic Programming
• Swarm Intelligence
• Reinforcement Learning
• Cyber Security
• Cognitive Computing
• Other Books You May Enjoy