Big Data Revolution

What farmers, doctors and insurance agents teach us about discovering big data patterns

Rob Thomas, Patrick McSharry

• Publisher: Wiley
• Year: 2015
• ISBN:
  9781118943724
• Edition: 1
• Topic: Technologie et ingénierie
• Subtopic: Ingénierie civile

Exploit the power and potential of Big Data to revolutionize business outcomes

Big Data Revolutionis a guide to improving performance, making better decisions, and transforming business through the effective use of Big Data. In this collaborative work by an IBM Vice President of Big Data Products and an Oxford Research Fellow, this book presents inside stories that demonstrate the power and potential of Big Data within the business realm. Readers are guided through tried-and-true methodologies for getting more out of data, and using it to the utmost advantage. This book describes the major trends emerging in the field, the pitfalls and triumphs being experienced, and the many considerations surrounding Big Data, all while guiding readers toward better decision making from the perspective of a data scientist.

Companies are generating data faster than ever before, and managing that data has become a major challenge. With the right strategy, Big Data can be a powerful tool for creating effective business solutions – but deep understanding is key when applying it to individual business needs.Big Data Revolutionprovides the insight executives need to incorporate Big Data into a better business strategy, improving outcomes with innovation and efficient use of technology.

• Examine the major emerging patterns in Big Data
• Consider the debate surrounding the ethical use of data
• Recognize patterns and improve personal and organizational performance
• Make more informed decisions with quantifiable results

In an information society, it is becoming increasingly important to make sense of data in an economically viable way. It can drive new revenue streams and give companies a competitive advantage, providing a way forward for businesses navigating an increasingly complex marketplace.Big Data Revolutionprovides expert insight on the tool that can revolutionize industries.

PART I: THE REVOLUTION STARTS NOW: 9 INDUSTRIES TRANSFORMING WITH DATA

Chapter 1: Transforming Farms with Data

Chapter 2: Why Doctors Will Have Math Degrees

Chapter 3: Revolutionizing Insurance: Why Actuaries Will become Data Scientists

Chapter 4: Personalizing Retail and Fashion

Chapter 5: Transforming Customer Relationships with Data

Chapter 6: Intelligent Machines

Chapter 7: Government and Society

Chapter 8: Corporate Sustainability

Chapter 9: Weather and Energy

CHAPTER 1: TRANSFORMING FARMS WITH DATA

CALIFORNIA, 2013

AS THE WHEELS came down on my cross-country flight, I prepared for our landing at San Francisco International Airport (SFO). Looking out the window, I could see the sprawl of Silicon Valley, the East Bay, and in the distance, the San Francisco skyline. It is hard to believe that I was here to explore agriculture in 2013, given that what I could see from the plane was mostly concrete, highways, and heavy construction.

Not too many miles away from SFO, I began to wind through the tight curves of back roads, making my way to the headquarters of a major agricultural producer. While I had never visited this company before, I had the opportunity to sit down with the executive team to explore the topic of big data in farming and agriculture.

I embraced the calm and serene scene, a far cry from the vibrancy of San Francisco and the rush of Silicon Valley. As we entered a conference room, the discussion turned to produce, as I asked, “Why is it that the strawberries that I bought last week taste so much better than the ones I bought the week before?” While I posed the question as a conversation starter, it became the crux of our discussion.

It seems that quality — and, more specifically, consistency of quality — is the foremost issue on the mind of major producers. I asked about the exquisite quality of produce in Japan. The executive team quickly noted that Japan achieves quality at the price of waste. Said another way, they keep only 10 percent of what a grower provides. This clarified the point in my mind that quality, consistency of quality, and eliminating waste create the three sides of a balanced triangle.

The conversation that followed revealed one significant consensus in the room: Weather alone impacts crop production and the consistency of crops. And since no one in the room knew how to change the weather, they believed that this was the way things would always be. I realized that by blaming the weather this team believed their future did not belong in their own hands but was controlled by the luck, or the misfortune, of each passing season.

BRIEF HISTORY OF FARMING

The evolution of farming in the developed world provides context to much of the conventional wisdom about farming that exists today. Dating back to the 1700s, farming has been defined by four eras:

• 1700s (Subsistence Farming): Farmers produced the minimum amount of food necessary to feed their families and have some in reserve for the cold winter months.
• 1800s (Farming for Profit): This era marked the transition from subsistence farming to for-profit farming. This is when the widespread use of barns began, for the purpose of storing tools, crops, and related equipment. These were called pioneer farms.
• Early 1900s (Power Farming): At this time, the “power” came in the form of 1,800-pound horses. The farmers used animals for plowing, planting, and transporting crops. The use of animal labor drove the first significant increase in crop productivity.
• Mid- to late 1900s (Machine Farming): Sparked by the Industrial Revolution, this era’s farmers relied on the automation of many of the tasks formerly done by hand or animal. The addition of machinery created tremendous gains in productivity and quality.

Each era represented a significant step forward, based on the introduction of new and tangible innovations: barns, tools, horses, or machines. The progress was physical in nature, as you could easily see the change happening on the farms. In each era, production and productivity increased, with the most significant increases in the latter part of the 20^th century.

Farm productivity over time

Through these stages, farming became more productive, but not necessarily more intelligent.

THE DATA ERA

The current era of farming is being driven by the application of data. It is less understood than previous eras because it is not necessarily physical in nature. It’s easy to look at a horse and understand quickly how it can make farm labor easier, but understanding how to use geospatial information is a different proposition. The advantage is driven by intangibles: knowledge, insight, decision making. Ultimately, data is the fertilizer for each of those intangibles.

A simple understanding of how a crop grows can aid in understanding the impact of data on farms. The basic idea is that a plant needs sunlight, nutrients from the soil, and water to grow into a healthy plant, through a process called photosynthesis. Healthy plants must keep cool through a process called transpiration (similar to how a human sweats when physically stressed). But, if a plant lacks the nutrients or conditioning to transpire, then its functions will start to break down, which leads to damage. Using data to improve farming is fundamentally about having the ability to monitor, control, and if necessary, alter these processes.

Today, according to the Environmental Protection Agency, there are 2.2 million farms in the United States and many more outside of the U.S. The average farm spends $110,000 per year on pest control, fertilizer, and related items to drive yield. The prescient way to improve profit and harvest yields across a vast territory requires better collection, use, and application of data.

POTATO FARMING

Potato farming can be exceedingly difficult, especially when attempted at a large scale with the goal of near perfect quality. The problem with potato farming is that the crop you are interested in is underground; therefore, producing a high-quality and high-yield potato crop depends on agronomic management during the growing process.

At the Talking Data South West Conference in 2013, Dr. Robert Allen, a Senior GIS Analyst at Landmark Information Group, highlighted the importance of data in potato farming, in his talk titled, “Using Smartphones to Improve Agronomic Decision Making in Potato Crops.” Dr. Allen makes the case that leveraging data that describes the growth and the maturation of a crop during the growing season is instrumental to a successful yield. Continuous insight, delivered throughout the growing season, may have a material impact on the productivity of a crop.

One of the key variables required for yield prediction, and needed to manage irrigation, is groundcover. Groundcover, which calculates the percentage of ground covered by green leaf, provides critical input in the agronomic management of potato crops. Measuring groundcover is not as simple as pulling out a measuring tape; it requires capturing imagery of potato crops and large-scale collection of data related to the images (the water balance in soil, etc.), and the data must be put in the hands of farm managers and agronomists so that they can actually do something about what the data is telling them. The goal is not to collect data, but to act on it.

Dr. Allen describes four considerations in potato farming related to using data:

• Time: Data needs to be collected at regular intervals and decisions need to be made in near-real time.
• Geography: These tend to be large-scale operations (10,000 to 20,000 acres), with fields distributed over large areas.
• Man power: Data is often collected by farm field assistants (not scientists) and must be distributed because decision makers tend to be remote from the field.
• Irrigation: Irrigation, while very expensive, is a primary factor in the maturation of a potato crop. Utilizing data to optimize the use of irrigation can lead to a productive crop, at the lowest possible cost.

These considerations led to a data collection and analysis solution called CanopyCheck. While it requires only a download from Apple’s App Store, it provides a rich data experience to compare groundcover and other related data to optimize the quality and yield of a potato crop.

The Landmark Information Group describes CanopyCheck (http://download.cnet.com/ios/landmark-information-group/3260-20_4-10094055.html) as

This app is for potato growers, using the CanopyCheck groundcover monitoring system, and captures accurate and reliable images of the potato crop which can be used to describe crop development. Each image is geo-located and labelled with farm and field information specified by the potato grower on the accompanying CanopyCheck website.

Conventional wisdom states that growing potatoes is easy: They don’t need sunlight, they do not need daily care, and by controlling the amount of water they receive, growing potatoes is a fairly simple process. However, as is often the case, conventional wisdom overlooks the art of the possible. In the case of potatoes, the application of data and agronomy can drive yield productivity up 30 to 50 percent, which is material in terms of the economics and the waste that is reduced.

PRECISION FARMING

Whether you strike up a conversation with a farmer in the 1800s, 1900s, or even in the early part of this century, they would highlight:

• Their growing strategy evolves each year.
• While the strategy evolves, their ability to execute improves each year, based on increased knowledge.

While this farming approach has been good enough for the better part of three centuries, the Data era ushers in the notion of precision farming. According to Tom Goddard, of the Conservation and Development Branch of Alberta Agriculture, Food and Rural Development, the key components of precision farming are:

• Yield monitoring: Track crop yield by time or distance, as well as distance and bushels per load, number of loads, and fields.
• Yield mapping: Global Positioning System (GPS) receivers, along with yield monitors, provide spatial coordinates, which can be used to map entire fields.
• Variable-rate fertilizer: Managing the application of a variety of fertilizer materials.
• Weed mapping: Mapping weeds using a computer connected to a GPS receiver while adjusting the planting strategy, as needed.
• Variable spraying: Once you know weed locations from weed mapping, spot control can be practiced.
• Topography and boundaries: Creating highly accurate topographic maps using a Differential Global Positioning System (DGPS). This data can be used to take action on yield maps.
• Salinity mapping: This is valuable in interpreting yield maps and weed maps, as well as tracking the salinity over a period of time.
• Guidance systems: Guidance systems, such as DGPS (accurate to a foot or less) are valuable for assessing fields.
• Records and analyses: Large data collection is necessary to store pertinent data assets, along with images and geospatial information. It is important that this information can be archived and retrieved for future use.

The extensive insight that can be gained by collecting each of these data points is potentially revolutionary. It evolves a process from instinctual to data-driven — which, as seen in the potato example, has a fundamental impact on yields and productivity.

The underlying assumption is that the tools and methodology for capturing farm data are available and utilized efficiently. This is a big assumption because many farms today are not set up to actively collect and capitalize on new data assets. Accordingly, the ability to capture farm data becomes the source of competitive advantage.

CAPTURING FARM DATA

It sounds easy. Collect data. Then use that data to deliver insights. But, for anyone who has been on a rural farm in the last decade, it is easier said than done. There are limitations that exist on many farms: lack of digital equipment, lack of skilled technology labor, poor distribution of electricity, and poorly defined processes. Because of these factors, each farmer must establish a new order of doing things to take advantage of the Data era.

The data landscape for farming consists of three primary inputs.

Data landscape for farming

• Sensing equipment: Mounted devices on machinery, in fields, or anywhere near crops could be designed to collect/stream data or to control the application of water, pesticides, etc. This could range from instrumented tractors for harvesting to devices to monitor crop transpiration. The evolution of machines to collect data on crops and soil has been dramatic. In the last decade alone, equipment has evolved from mechanical-only to a combination of mechanical and digital technology. This change has been expedited by early insights that even small adjustments in planting depth or spacing can have huge impact on yields. So, while today the sensing equipment is largely a digitized version of common farm machines, the future will see a marked advancement in machines. Drones, driverless tractors, and other innovations will become commonplace.
• Global Positioning System (GPS): GPS provides the ability to pinpoint location accuracy within one meter. While GPS first emerged for automobiles in the early 1990s in places like Japan, it has just now become common in all automobiles. Farming equipment, as you may expect, has been even a step further behind, with the wide use of GPS just accelerating in the last decade.
• Geographic Information System (GIS): GIS assesses changes in the environment, tracks the spread of disease, as well as understanding where soil is moist, eroded, or has experienced similar c...