Big Data Velocity

10 Key excerpts on "Big Data Velocity"

• 

  eBook - ePub

  Big Data Computing

  A Guide for Business and Technology Managers

  • Vivek Kale(Author)
  • 2016(Publication Date)
  • Chapman and Hall/CRC

    (Publisher)

  9.1.1.2. Data Velocity

  The business models adopted by Amazon, Facebook, Yahoo!, and Google, which became the defacto business models for most web-based companies, operate on the fact that by tracking customer clicks and navigations on the website, you can deliver personalized browsing and shopping experiences. In this process of clickstreams, there are millions of clicks gathered from users at every second, amounting to large volumes of data. This data can be processed, segmented, and modeled to study population behaviors based on time of day, geography, advertisement effectiveness, click behavior, and guided navigation response. The result sets of these models can be stored to create a better experience for the next set of clicks exhibiting similar behaviors. The velocity of data produced by user clicks on any website today is a prime example for Big Data Velocity.

  Real-time data and streaming data are accumulated by the likes of Twitter and Facebook at a very high velocity. Velocity is helpful in detecting trends among people that are tweeting a million tweets every 3 minutes. Processing of streaming data for analysis also involves the velocity dimension. Similarly, high velocity is attributed to data associated with the typical speed of transactions on stock exchanges; this speed reaches billions of transactions per day on certain days. If these transactions must be processed to detect potential fraud or billions of call records on cell phones daily must be processed to detect malicious activity, we are dealing with the velocity dimension.

  The most popular way to share pictures, music, and data today is via mobile devices. The sheer volume of data that is transmitted by mobile networks provides insights to the providers on the performance of their network, the amount of data processed at each tower, the time of day, the associated geographies, user demographics, location, latencies, and much more. The velocity of data movement is unpredictable, and sometimes can cause a network to crash. The data movement and its study have enabled mobile service providers to improve the QoS (quality of service), and associating this data with social media inputs has enabled insights into competitive intelligence.

  Sign up to read

  Learn more about book
• 

  No longer available |Learn more

  Big Data Architect's Handbook

  A guide to building proficiency in tools and systems used by leading big data experts

  • Syed Muhammad Fahad Akhtar(Author)
  • 2018(Publication Date)
  • Packt Publishing

    (Publisher)

  In the last two years, the amount of data generated is equal to 90% of the data ever created. The world's data is doubling every 1.2 years. One survey states that 40 zettabytes of data will be created by 2020.

  Not so long ago, the generation of such massive amount of data was considered to be a problem as the storage cost was very high. But now, as the storage cost is decreasing, it is no longer a problem. Also, solutions such as Hadoop and different algorithms that help in ingesting and processing this massive amount of data make it even appear resourceful.

  The second characteristic of big data is velocity. Let's find out what this is. Passage contains an image

  Velocity

  Velocity is the rate at which the data is being generated, or how fast the data is coming in. In simpler words, we can call it data in motion. Imagine the amount of data Facebook, YouTube, or any social networking site is receiving per day. They have to store it, process it, and somehow later be able to retrieve it. Here are a few examples of how quickly data is increasing:

  • The New York stock exchange captures 1 TB of data during each trading session.
  • 120 hours of videos are being uploaded to YouTube every minute.
  • Data generated by modern cars; they have almost 100 sensors to monitor each item from fuel and tire pressure to surrounding obstacles.
  • 200 million emails are sent every minute.

  If we take the example of social media trends, more data means more revealing information about groups of people in different territories: Velocity at which the data is being generated

  The preceding chart shows the amount of time users are spending on the popular social networking websites. Imagine the frequency of data being generated based on these user activities. This is just a glimpse of what's happening out there.

  Another dimension of velocity is the period of time during which data will make sense and be valuable. Will it age and lose value over time, or will it be permanently valuable? This analysis is also very important because if the data ages and loses value over time, then maybe over time it will mislead you.

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Guide to Cloud Computing for Business and Technology Managers

  From Distributed Computing to Cloudware Applications

  • Vivek Kale(Author)
  • 2014(Publication Date)
  • Chapman and Hall/CRC

    (Publisher)

  In this process of clickstreams, there are millions of clicks gathered from users at every second, amounting to large volumes of data. These data can be processed, segmented, and modeled to study population behaviors based on time of day, geography, advertisement effectiveness, click behavior, and guided navigation response. The result sets of these models can be stored to create a better experience for the next set of clicks exhibiting similar behav-iors. The velocity of data produced by user clicks on any website today is a prime example for Big Data Velocity. TABLE 21.1 Scale of Data Size of Data Scale of Data 1000 megabytes 1 gigabyte (GB) 1000 gigabytes 1 terabyte (TB) 1000 terabytes 1 petabyte (PB) 1000 petabytes 1 exabyte (EB) 1000 exabytes 1 zettabyte (ZB) 1000 zettabytes 1 yottabyte (YB) 444 Guide to Cloud Computing for Business and Technology Managers The most popular way to share pictures, music, and data today is via mobile devices. The sheer volume of data that is transmitted by mobile networks provides insights to the providers on the performance of their network, the amount of data processed at each tower, the time of day, the associated geog-raphies, user demographics, location, latencies, and much more. The velocity of data movement is unpredictable and sometimes can cause a network to crash. The data movement and its study have enabled mobile service provid-ers to improve the QoS (quality of service), and associating these data with social media inputs has enabled insights into competitive intelligence. The list of features for handling data velocity included the following: • System must be elastic for handling data velocity along with volume. • System must scale up and scale down as needed without increasing costs. • System must be able to process data across the infrastructure in the least processing time. • System throughput should remain stable independent of data velocity. • System should be able to process data on a distributed platform.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Software Architecture for Big Data and the Cloud

  • Ivan Mistrik, Rami Bahsoon, Nour Ali, Maritta Heisel, Bruce Maxim(Authors)
  • 2017(Publication Date)
  • Morgan Kaufmann

    (Publisher)

  The accepted definition of big data is the digital analysis of datasets to extract insights, correlations and causations, and value from data. Different groups have come up with different “Vs” to attempt to formalize the definition of the big aspect of this phenomenon. The 3 Vs definition of big data, by Doug Laney for Gartner, states that if it has Volume, Variety and Velocity then the data can be considered big [39]. Bernard Marr, in his book Big Data, adds Veracity (or validity) and Value to the original list to create the 5 V's of big data [50]. With Volatility and Variability, Visibility, and Visualization added in some combination to the list by different authors there is now a 7 Vs definition of what constitutes big data [3, 46, 54, 60]. Using sales and advertising as a basis the Vs based definition can be explained as: • Volume – With more data being collected, logged, and generated organizations need larger storage to retain this information and larger compute to process it. • Velocity – Through online transactions and interactions the rate at which the Volume is being created vastly exceeds data generated from person-to-person interactions. Online systems are also expected to react in a timely manner meaning that the data needs to be processed as quickly as it gets ingested. • Variety – A digital transaction gives more than just a sale, even a customer browsing certain sections of an online store is valuable information, whether or not a sale is made. In an online transaction, person A buying Object-X is not the only information that can be extracted. Socio-economic, demographic, and consumer journey information can all be collected to improve future sales and advertising. The problem becomes more complex within the inclusion of data from traditional and social media. • Veracity – Large volumes of disparate data being ingested at high speed are only useful if the information is correct

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Engaging Customers Using Big Data

  How Marketing Analytics Are Transforming Business

  • Arvind Sathi(Author)
  • 2017(Publication Date)
  • Palgrave Macmillan

    (Publisher)

  There are two aspects to velocity, one representing the throughput of data and the other representing latency. Let me start with throughput, which represents the data moving in the pipes. The amount of global mobile data is growing at a 78 percent compounded growth rate, and is expected to reach 10.8 exabytes per month in 2016 5 as consumers share more pictures and videos. To analyze this data, the corporate analytics infrastructure is seeking bigger pipes and massively parallel process- ing. Latency is the other measure of velocity. Analytics used to be a “store and report” environment in which reporting typically contained data as of yesterday—popularly represented as “D-1.” Now, the analyt- ics is increasingly being embedded in business processes using data- in-motion with reduced latency. For example, Turn (www.turn.com) is conducting its analytics in ten milliseconds to place advertisements in online advertising platforms. 6 These flows no longer represent structured data. Conversations, documents, and web pages are good examples of unstructured data. Some of the data, such as that coming from telecom networks is some- what structured, but carries such a large variety of formats that it is almost unstructured. All this leads to a requirement for dealing with high variety. In the 1990s, as data warehouse technology was intro- duced, the initial push was to create meta-models to represent all the data in one standard format. The data was compiled from a variety of sources and transformed using ETL (extract, transform, load) or ELT (extract the data and load it in the warehouse, then transform it inside the warehouse). The basic premise was a narrow variety and structured content. Big data has significantly expanded our horizons, enabled by new data integration and analytics technologies. A number of call cen- ter analytics solutions are seeking analysis of call center conversations and their correlation with emails, trouble tickets, and social media blogs.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Big Data Analysis for Green Computing

  Concepts and Applications

  • Rohit Sharma, Dilip Kumar Sharma, Dhowmya Bhatt, Binh Thai Pham, Rohit Sharma, Dilip Kumar Sharma, Dhowmya Bhatt, Binh Thai Pham(Authors)
  • 2021(Publication Date)
  • CRC Press

    (Publisher)

  Big data are used to define the enormous volume of data that cannot be stored, processed, or analyzed using traditional database technologies. These data or data sets are too large and complex for the existing traditional database. The idea of big data is vague and includes extensive procedures to recognize and make an interpretation of the information into new bits of knowledge. Although the term big data is available from the last century; however, its true meaning has been recognized after social media has become popular. Thus, one can say that the term is relatively new to the IT industry and organizations. However, there are several instances where the researchers have used and utilized the term in their literature.

  The authors in [9 ] defined a large volume of scientific data as big data required for visualization. Several authors have defined big data in different ways. One of the earliest was given by Gartner in 2001 [1 ]. The Gartner definition has no keyword as “big data”; however, he defines the term with three Vs: volume, velocity, and variety. He has discussed the increasing rate and size of data. This definition given by Gartner is later adopted by various agencies and authors such as NIST [10 ], Gartner himself in 2012 [11 ], and later IBM [12 ], and others include a fourth V to original three Vs: veracity. The authors in [13 ] explained the term big data as “the volume of data that is beyond the current technology’s capability to efficiently store, manage, and process”. The authors in [14 ] and [15 ] used Gartner’s classification of 2001 of three Vs: volume, variety, and velocity to define big data (Figure 3.1 ).

  FIGURE 3.1

  Three Vs of big data.

  So based on the discussion, the term big data can be defined as a high volume of a variety of data that are generated at a rapid speed, and the traditional database system is not fully functional to store, process, and analyze that in real time. Let us look into the three Vs that are defined by various authors in their work.

  1. Volume is the measure of information created from a variety of sources that keep on growing. The major benefit of such a large collection of information is that it helps in decision making by identifying hidden patterns with the help of data analytics. Having more parameters let us say 200 to forecast weather will be able to predict weather better as compared to forecasting with 4–6 parameters. The volume in big data refers to the size of zettabytes (ZB – 10,007 bytes) or yottabyte (YB – 10,008 bytes). Thus, it becomes a huge challenge for the current infrastructure to store this amount of data. Most companies like to put their old data in archives or logs form, i.e., in an offline mode. But the disadvantage of this is that the data are not available for processing. Thus, it requires a scalable and distributed storage that is being offered by cloud as cloud storage either in the form of object, file, or block storage.
     Assuming that the enormous volume of the data cannot be processed by the traditional database, the options left for processing are either breaking the data into chunks for massively parallel processing frameworks such as Apache Hadoop or database like Greenplum. Using the data warehouse or database evolves the predefined schemas to be entered into the system, given the other Vs – variety: this is again not feasible for big data. Apache Hadoop does not place any such condition on the structure of data and can process it without a predefined structure. For storing data, the Hadoop framework used its own distributed file system known as Hadoop Distributed File System or HDFS. When data are required by any node, HDFS provides that data to the node. A typical Hadoop framework has three steps for storing data [16

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  The Human Element of Big Data

  Issues, Analytics, and Performance

  • Geetam S. Tomar, Narendra S. Chaudhari, Robin Singh Bhadoria, Ganesh Chandra Deka(Authors)
  • 2016(Publication Date)
  • Chapman and Hall/CRC

    (Publisher)

  The speed of the incoming data has reduced to fractions of seconds. This high velocity data represent Big Data. • Variety—The term variety describes the range of data types and sources. In gen-eral, most organizations use the following type of data formats such as database, excel, and CSV, which can be stored in a simple text file. However, sometimes the data may not be in the prescribed format as we assume; it may be in the form of audio, video, SMS, pdf, or something we might have not thought about it. This can be overcome by developing data storage system that can store varieties of data. • Value—The term value describes the worth of the data being extracted. Having endless amounts of data is one thing, but unless it can be turned into value it is useless. While there is a strong connection between data and insights, this does not always mean there is value in Big Data. The most important point to be consid-ered is to understand the costs and benefits of collecting and analyzing the huge data. This value of data represents Big Data. 10 V’s of Big Data Visualization Volume Velocity Variety Value Veracity Validity Variability Viscosity Virality FIGURE 16.1 10 V’s of Big Data. 307 Big Data Architecture for Climate Change and Disease Dynamics • Veracity—Veracity doesn’t mean data quality; it’s about data understandability. In other words, it assumes that data is being stored and mined properly to make it pertinent to the problem being analyzed. In order to use effective information from Big Data, the organization should perform data clean and process it to pre-vent “dirty data” from accumulating in the systems. • Validity—Validity is similar to veracity. It checks whether the data is correct and accurate for the intended use. Clearly, valid data is the key to making the right decisions in the future.

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Big Data and Social Science

  • Sudha Menon, University of Kerala, India(Authors)
  • 2019(Publication Date)
  • Society Publishing

    (Publisher)

  it is also described as a cluster of distinct database stores mainly in online shopping and web browsing. As big data develops into the field of ML and artificial intelligence (AI) , velocity is particularly beneficial in context to the speed that different algorithms can ingest, compare, and analyze the sources of data. Introduction to Big Data 11 1.6.1. Urban Planning Big data is not a science fiction; however, it feels like it. Proper and accurate urban planning is becoming more complex with a great number of systems competing for different resources such as energy, utilities, housing, infrastructure, transportation, etc. There are huge amounts of data related to cities, their residents and how they are using their spaces. This means that urban planners are required to be expert at using big data. Several cities are now using big data from the IoT in order to transform their municipalities into smart cities. For instance, London make use of Big Data to manage the waste, reduce the cost and enhance the quality of living as well as working in large cities. Utilization of Big data in urban planning helps in tackling issues of parking, pollutions, and consumption of energy (Figure 1.4). Figure 1.4: Big data can help in urban planning. Source: https://pixabay.com/photos/city-urban-urban-planning-build-ing-1487759/. 1.6.2. Protecting the Environment Big data can help to save the planet. A delicate issue of deforestation removes trees and habitat for several other species of plant and animal. Big data provides alternate solutions to the cutting of tree so that the carbon footprint can be decreased. High-tech data processing, satellite images and crowdsourcing can easily provide real-time data on the forest of the world. Big Data and Social Science 12 This is right, creation of big data maps of the forest present all around the world. Big data provides the chance to preserve endangered species as well as to mitigate poaching.

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Data analysis and Information processing

  • Jovan Pehcevski(Author)
  • 2023(Publication Date)
  • Arcler Press

    (Publisher)

  Characteristics of Big Data The most prominent features of big data are characterized as Vs. The first three Vs of Big data are Volume for huge data amount, Variety for different Big Data for Organizations: A Review 191 types of data, and Velocity for different data rate required by different kinds of systems [6]. Volume: When the scale of the data surpass the traditional store or technique, these volume of data can be generally labeled as the big data volume. Based on the types of organization, the amount of data volume can vary from one place to another from gigabytes, terabytes, petabytes, etc. [1]. Volume is the original characteristic for the emergence of big data. Variety: Include structured data defined in specific type and structure (e.g. string, numeric, etc. data types which can be found in most RDBMS databases), semi-structured data which has no specific type but have some defined structure (e.g. XML tags, location data), unstructured data with no structure (e.g. audio, voice, etc. ) which their structures have to be discovered yet [7], and multi-structured data which include all these structured, semi-structured and unstructured features [7] [8]. Variety comes from the complexity of data from different information systems of target organization. Velocity: Velocity means the rate of data required by the application systems based on the target organization domain. The velocity of big data can be considered in increasing order as batch, near real-time, real-time and stream [7]. The bigger data volume, the more challenges will likely velocity face. Velocity the one of the most difficult characteristics in big data to handle [8]. As more and more organizations are trying to use big data, big data Vs characteristics become to appear one after another such as value, veracity and validity. Value mean that data retrieved from big data must support the objective of the target organization and should create a surplus value for the organization [7].

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Communication, Management and Information Technology

  International Conference on Communciation, Management and Information Technology (ICCMIT 2016, Cosenza, Italy, 26-29 April 2016)

  • Marcelo Sampaio de Alencar(Author)
  • 2016(Publication Date)
  • CRC Press

    (Publisher)

  The goals of this ini-tiative were to develop and improve technologies needed to collect, store, manage, and analyze this big data, to use these technologies to accelerate the pace of knowledge discovery in science and engineering fields, improve national security, and transform teaching and learning, and to expand the workforce required to develop and use big data technologies [9]. According to McKinsey [10] the term big data is used to refer to datasets whose size is beyond the capability of existing database software tools to capture, store, manage and analyze within a toler-able amount of time. However, there is no single definition of big data. O’Reilly [11] defines big data as “data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the structures of existing database architectures. To gain value from this data, there must be an alternative way to proc-ess it”. As seen from the above definitions, the volume of data is not the only characteristic of big data. In fact, big data has three major characteristics (known as 3V’s), shown in Figure 2, which were first defined by Doug Laney in 2001 [12]. • Data volume (i.e. the size of data) is the primary attribute of big data. The size of data could reach terabytes (TB, 10 12 B), petabytes (PB, 10 15 B), exabytes (EB, 10 18 B), zettabytes (ZB, 10 21 B) and more. For example, Facebook reached more than 8 billion video views per day in September 2015 [13]. • Variety refers to the fact that big data can come from different data sources in various formats and structures. These data sources are divided into three types: structured, semi-structured and unstructured data [14]. Structured data is described as data that follows a fixed schema. An example of this type is a relational database sys-tem. Semi-structured data is a type of structured data, but it doesn’t have a rigid structure [15]. Its structure may change rapidly or unpredict-ably [15].

  Sign up to read

  Learn more about book