Apache Ignite Quick Start Guide

Distributed data caching and processing made easy

Sujoy Acharya

• Publisher: Packt Publishing
• Year: 2018
• Print ISBN: 9781789347531
• eBook ISBN: 9781789344066
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Modelling & Design
• Index: Computer Science

Build efficient, high-performance & scalable systems to process large volumes of data with Apache Ignite

Key Features

• Understand Apache Ignite's in-memory technology
• Create High-Performance app components with Ignite
• Build a real-time data streaming and complex event processing system

Book Description

Apache Ignite is a distributed in-memory platform designed to scale and process large volume of data. It can be integrated with microservices as well as monolithic systems, and can be used as a scalable, highly available and performant deployment platform for microservices. This book will teach you to use Apache Ignite for building a high-performance, scalable, highly available system architecture with data integrity.

The book takes you through the basics of Apache Ignite and in-memory technologies. You will learn about installation and clustering Ignite nodes, caching topologies, and various caching strategies, such as cache aside, read and write through, and write behind. Next, you will delve into detailed aspects of Ignite's data grid: web session clustering and querying data.

You will learn how to process large volumes of data using compute grid and Ignite's map-reduce and executor service. You will learn about the memory architecture of Apache Ignite and monitoring memory and caches. You will use Ignite for complex event processing, event streaming, and the time-series predictions of opportunities and threats. Additionally, you will go through off-heap and on-heap caching, swapping, and native and Spring framework integration with Apache Ignite.

By the end of this book, you will be confident with all the features of Apache Ignite 2.x that can be used to build a high-performance system architecture.

What you will learn

• Use Apache Ignite's data grid and implement web session clustering
• Gain high performance and linear scalability with in-memory distributed data processing
• Create a microservice on top of Apache Ignite that can scale and perform
• Perform ACID-compliant CRUD operations on an Ignite cache
• Retrieve data from Apache Ignite's data grid using SQL, Scan and Lucene Text query
• Explore complex event processing concepts and event streaming
• Integrate your Ignite app with the Spring framework

Who this book is for

The book is for Big Data professionals who want to learn the essentials of Apache Ignite. Prior experience in Java is necessary.

Exploring the Compute Grid and Query API

In Chapter 3, Working with Data Grids, we learned to populate and fetch Apache Ignite cache entries using the key-value pair API. This data access API is really fast, but it lacks querying capability. The next section explains the Apache Ignite Query API, used to efficiently manipulate cached entries.

This chapter covers two important components of Apache Ignite architecture: Apache Ignite Query API and distributed computing using the Apache Ignite Compute Grid. The following topics will be covered:

• Query API
• Compute grid

Query API

Apache Ignite offers an elegant query API with the following components: predicate-based ScanQuery, ANSI 99-compliant SQL query, and Lucene index-based text query. In this section, we will examine the query API.

ScanQuery

Apache Ignite's key-value pair API is used to store objects in a cache and retrieve values using keys. Apache Ignite's query API lets us query objects using expressions. The ScanQuery API allows us to execute distributed queries over cache objects. We are going to create a cache, populate it with a collection of objects, and then use ScanQuery to retrieve them. Follow these steps to try out the ScanQuery API:

1. Create a class, Player, with the following members:

 public class Player implements Serializable {
 private static final long serialVersionUID = 1L;
 private Long id;
 private String name;
 private String team;
 private double salary;
 public Player(Long id, String name, String team, double salary) {
 this.id = id;
 this.name = name;
 this.team = team;
 this.salary = salary;
 }
 @Override
 public String toString() {
 return "Player [id=" + id + ", name=" + name + ", team=" + team + 
 ", salary=" + salary + "]";
 }
 //Getters/setters here
 }

2. Add a class, ScanQueryTest, create a cache with the name Player_Scan_Cache, and populate it with some players:

 IgniteConfiguration cfg = new IgniteConfiguration();
 cfg.setPeerClassLoadingEnabled(true);
 try (Ignite ignite = Ignition.start(cfg)) {
 IgniteCache<Long, Player> playerCache = 
 Ignition.ignite().getOrCreateCache(PLAYER_SCAN_CACHE);
 long id = 1l;
 playerCache.put(id, new Player(id++, "Leo Messi", 
 "Barcelona", 996999995.00d));
 playerCache.put(id, new Player(id++, "Christiano Ronaldo", 
 "Juventus", 2000000.00d));
 playerCache.put(id, new Player(id++, "Paul Pogba", 
 "Manchester United", 1000000.00d));
 playerCache.put(id, new Player(id++, "Neymar", "PSG", 
 99699999.00d));
 playerCache.put(id, new Player(id++, "Luis Suárez", 
 "Barcelona", 578699.00d));

3. Now, query the cache using ScanQuery. IgniteCache has a method to pass Query and it returns QueryCursor. ScanQuery is an implementation of Query; it takes IgniteBiPredicate as an argument. IgniteBiPredicate takes two parameters and returns a Boolean. We are going to use a Java 8 lambda expression to represent IgniteBiPredicate. The i represents the key of the cache and p is the value or Player. Our IgniteBiPredicate returns true only if any player stored in the cache qualifies to the expression player.getTeam() EQ Barcelona. The result is returned as a QueryCursor; it stores all qualified entries (key-value pairs). We are going to use a Java 8 lambda to loop through the entries and print their details:

 System.out.println("Barcelona Soccer Players");
 QueryCursor<Entry<Long, Player>> barcelonaPlayersCursor = 
 playerCache
 .query(new ScanQuery<Long, Player>((i, p) -> 
 p.getTeam().equalsIgnoreCase("Barcelona")));

 barcelonaPlayersCursor.forEach(e -> {
 System.out.println(e.getValue());
 });

4. Fetch all players who earn more than 1,000,000 USD . The query could be simple (i, p) -> p.getSalary() > 1000000:

 System.out.println("Rich Soccer Players");
 QueryCursor<Entry<Long, Player>> richPlayers = playerCache
 .query(new ScanQuery<Long, Player>((i, p) -> p.getSalary() > 
 1000000));
 
 richPlayers.forEach(e -> {
 System.out.println(e.getValue());
 });

When we run the previous program, it prints the following output:

The next section explores text-based full search.

TextQuery

The TextQuery API allows us to run full text search on stored objects. The ScanQuery goes over each cache entries and apply the predicate, which is not a very efficient way to query objects. The TextQuery works on Lucene indexes; Elasticsearch and Apache Solr use Lucene for indexing text. It is always advisable to use indexes for querying entries, but one drawback is an index itself takes up space and slows down the data modification (create and update) as every time you modify an entry, the index needs to be rebuilt.

You need to define the metadata to tell Apache Ignite which fields to be indexed. The @QueryTextField annotation enables indexing. But, your cache configuration also needs to enable indexing by setting the setIndexedTypes.

Let's explore the TextQuery API. These are the steps:

1. Modify the Player class; annotate the name and team fields with the @QueryTextField annotation:

 @QueryTextField()
 private String name;
 @QueryTextField
 private String team;

2. Add a class, TextQueryTest, and configure the cache:

 private static final String PLAYER_TEXT_CACHE = 
 "Player_Text_Cache";

 public static void main(String[] args) {
 IgniteConfiguration cfg = new IgniteConfiguration();
 cfg.setPeerClassLoadingEnabled(true);
 CacheConfiguration<Long, Player> playerCacheConfig = new 
 CacheConfiguration<>();
 playerCacheConfig.setName(PLAYER_TEXT_CACHE);
 playerCacheConfig.setIndexedTypes(Long.class, Player.class);
 cfg.setCacheConfiguration(playerCacheConfig);

3. Populate the cache with a few players:

 try (Ignite ignite = Ignition.start(cfg)) ...

Table of contents

• Title Page
• Copyright and Credits
• About Packt
• Contributors
• Preface
• Getting Familiar with Apache Ignite
• Understanding the Topologies and Caching Strategies
• Working with Data Grids
• Exploring the Compute Grid and Query API
• Building MicroServices with Service Grid
• Sharpening Ignite Skills
• Deploying To Production
• Other Books You May Enjoy