- 176 pages
- English
- ePUB (mobile friendly)
- Available on iOS & Android
eBook - ePub
YARN Essentials
Trusted by 375,005 students
Access to over 1 million titles for a fair monthly price.
Study more efficiently using our study tools.
Information
YARN Essentials
Table of Contents
YARN Essentials
Credits
About the Authors
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
1. Need for YARN
The redesign idea
Limitations of the classical MapReduce or Hadoop 1.x
YARN as the modern operating system of Hadoop
What are the design goals for YARN
Summary
2. YARN Architecture
Core components of YARN architecture
ResourceManager
ApplicationMaster (AM)
NodeManager (NM)
YARN scheduler policies
The FIFO (First In First Out) scheduler
The fair scheduler
The capacity scheduler
Recent developments in YARN architecture
Summary
3. YARN Installation
Single-node installation
Prerequisites
Platform
Software
Starting with the installation
The standalone mode (local mode)
The pseudo-distributed mode
The fully-distributed mode
HistoryServer
Slave files
Operating Hadoop and YARN clusters
Starting Hadoop and YARN clusters
Stopping Hadoop and YARN clusters
Web interfaces of the Ecosystem
Summary
4. YARN and Hadoop Ecosystems
The Hadoop 2 release
A short introduction to Hadoop 1.x and MRv1
MRv1 versus MRv2
Understanding where YARN fits into Hadoop
Old and new MapReduce APIs
Backward compatibility of MRv2 APIs
Binary compatibility of org.apache.hadoop.mapred APIs
Source compatibility of org.apache.hadoop.mapred APIs
Practical examples of MRv1 and MRv2
Preparing the input file(s)
Running the job
Result
Summary
5. YARN Administration
Container allocation
Container allocation to the application
Container configurations
YARN scheduling policies
The FIFO (First In First Out) scheduler
The FIFO (First In First Out) scheduler
The capacity scheduler
Capacity scheduler configurations
The fair scheduler
Fair scheduler configurations
YARN multitenancy application support
Administration of YARN
Administrative tools
Adding and removing nodes from a YARN cluster
Administrating YARN jobs
MapReduce job configurations
YARN log management
YARN web user interface
Summary
6. Developing and Running a Simple YARN Application
Running sample examples on YARN
Running a sample Pi example
Monitoring YARN applications with web GUI
YARN's MapReduce support
The MapReduce ApplicationMaster
Example YARN MapReduce settings
YARN's compatibility with MapReduce applications
Developing YARN applications
The YARN application workflow
Writing the YARN client
Writing the YARN ApplicationMaster
Responsibilities of the ApplicationMaster
Summary
7. YARN Frameworks
Apache Samza
Writing a Kafka producer
Writing the hello-samza project
Starting a grid
Storm-YARN
Prerequisites
Hadoop YARN should be installed
Apache ZooKeeper should be installed
Setting up Storm-YARN
Getting the storm.yaml configuration of the launched Storm cluster
Building and running Storm-Starter examples
Apache Spark
Why run on YARN?
Apache Tez
Apache Giraph
HOYA (HBase on YARN)
KOYA (Kafka on YARN)
Summary
8. Failures in YARN
ResourceManager failures
ApplicationMaster failures
NodeManager failures
Container failures
Hardware Failures
Summary
9. YARN – Alternative Solutions
Mesos
Omega
Corona
Summary
10. YARN – Future and Support
What YARN means to the big data industry
Journey – present and future
Present on-going features
Future features
YARN-supported frameworks
Summary
Index
YARN Essentials
Copyright © 2015 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: February 2015
Production reference: 1190215
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78439-173-7
www.packtpub.com
Credits
Authors
Amol Fasale
Nirmal Kumar
Reviewers
Lakshmi Narasimhan
Swapnil Salunkhe
Jenny (Xiao) Zhang
Commissioning Editor
Taron Pereira
Acquisition Editor
James Jones
Content Development Editor
Arwa Manasawala
Technical Editor
Indrajit A. Das
Copy Editors
Karuna Narayanan
Laxmi Subramanian
Project Coordinator
Purav Motiwalla
Proofreaders
Safis Editing
Maria Gould
Indexer
Priya Sane
Graphics
Sheetal Aute
Valentina D'silva
Abhinash Sahu
Production Coordinator
Shantanu N. Zagade
Cover Work
Shantanu N. Zagade
About the Authors
Amol Fasale has more than 4 years of industry experience actively working in the fields of big data and distributed computing; he is also an active blogger in and contributor to the open source community. Amol works as a senior data system engineer at MakeMyTrip.com, a very well-known travel and hospitality portal in India, responsible for real-time personalization of online user experience with Apache Kafka, Apache Storm, Apache Hadoop, and many more. Also, Amol has active hands-on experience in Java/J2EE, Spring Frameworks, Python, machine learning, Hadoop framework components, SQL, NoSQL, and graph databases.
You can follow Amol on Twitter at
@amolfasale or on LinkedIn. Amol is very active on social media. You can catch him online for any technical assistance; he would be happy to help.Amol has completed his bachelor's ...
Table of contents
- YARN Essentials
Frequently asked questions
Yes, you can cancel anytime from the Subscription tab in your account settings on the Perlego website. Your subscription will stay active until the end of your current billing period. Learn how to cancel your subscription
No, books cannot be downloaded as external files, such as PDFs, for use outside of Perlego. However, you can download books within the Perlego app for offline reading on mobile or tablet. Learn how to download books offline
Perlego offers two plans: Essential and Complete
- Essential is ideal for learners and professionals who enjoy exploring a wide range of subjects. Access the Essential Library with 800,000+ trusted titles and best-sellers across business, personal growth, and the humanities. Includes unlimited reading time and Standard Read Aloud voice.
- Complete: Perfect for advanced learners and researchers needing full, unrestricted access. Unlock 1.4M+ books across hundreds of subjects, including academic and specialized titles. The Complete Plan also includes advanced features like Premium Read Aloud and Research Assistant.
We are an online textbook subscription service, where you can get access to an entire online library for less than the price of a single book per month. With over 1 million books across 990+ topics, we’ve got you covered! Learn about our mission
Look out for the read-aloud symbol on your next book to see if you can listen to it. The read-aloud tool reads text aloud for you, highlighting the text as it is being read. You can pause it, speed it up and slow it down. Learn more about Read Aloud
Yes! You can use the Perlego app on both iOS and Android devices to read anytime, anywhere — even offline. Perfect for commutes or when you’re on the go.
Please note we cannot support devices running on iOS 13 and Android 7 or earlier. Learn more about using the app
Please note we cannot support devices running on iOS 13 and Android 7 or earlier. Learn more about using the app
Yes, you can access YARN Essentials by Amol Fasale, Nirmal Kumar in PDF and/or ePUB format, as well as other popular books in Business & Business Intelligence. We have over one million books available in our catalogue for you to explore.