Big Data Challenges
Big data challenges refer to the difficulties associated with managing, analyzing, and extracting valuable insights from large and complex datasets. These challenges include issues related to data storage, processing, analysis, and privacy. Addressing big data challenges often requires advanced technologies and techniques such as distributed computing, machine learning, and data visualization.
6 Key excerpts on "Big Data Challenges"
eBook - ePubRecent Trends in Communication and Electronics
Proceedings of the International Conference on Recent Trends in Communication and Electronics (ICCE-2020), Ghaziabad, India, 28-29 November, 2020
- Sanjay Sharma, Astik Biswas, Brajesh Kumar Kaushik, Vibhav Sachan, Sanjay Sharma, Astik Biswas, Brajesh Kumar Kaushik, Vibhav Sachan(Authors)
- 2021(Publication Date)
- CRC Press(Publisher)
...As far as the business and consumer emails (both sent and received) is concerned, it was 293 billion in 2019, and is expected to grow to over 347 billion by the end of 2023 [3]. Facebook Statistics –As per the latest update about Facebook statistics, it is found that Worldwide, there are over 2.38 billion MAUs for December 2019[5]. 4 Challenges Of Big Data In recent years, there have been many applications of big data in the fields of health care, biochemistry, retail, and other interdisciplinary scientific researches. Not only this, big data has contributed a lot in web-based applications such as social computing, internet text and documents, and internet search indexing. Keeping these advantages of big data in mind, we can say that knowledge processing can take many benefits from big data. However, there are also many challenges which follow these opportunities. The various challenges that are being faced in the research of big data includes: Storing and analyzing big data First and foremost, challenge in managing big data is to find out the means to store and analyze big data. In recent times, bulk of data is created and stored every minute. To store this data, we need to have proper storage medium with high input/output speed. Data must be accessible easily from these mediums so that knowledge discovery becomes easy for further analysis.The conventional computing methods in data mining fail to meet the expectations when it comes to the processing and analysis of huge and fast changing data in big data. Traditional computing methods like machine learning works fine with only small quantity of data. Computational Complexities Not only the quantity but the quality of data also poses a challenge when we talk about analyzing the data in big data. The data types, structures and patterns in big data are so complex that the representation, understanding and computation become very difficult. This results in an increase in computational complexity...
eBook - ePubBig Data Analytics
Harnessing Data for New Business Models
- Soraya Sedkaoui, Mounia Khelfaoui, Nadjat Kadi, Soraya Sedkaoui, Mounia Khelfaoui, Nadjat Kadi(Authors)
- 2021(Publication Date)
- Apple Academic Press(Publisher)
...In addition to data quality, where data are unreliable, and incomplete, bad data, in the USA for example, cost 600 billion dollars each year. Moreover, analyzing a great quantity of data needs powerful algorithms and IT tools. Security and privacy, where safety is another challenge, given the huge amount of data; this includes user authentication, user-dependent access restriction, log data access history, proper use of data encryption, etc. Lack of talent, there are a lot of big data projects in the major organizations, but the presence of a talented team of developers, data scientists, and analysts with enough knowledge in the field represents a challenge [9]. Prohibitive costs, where it is not enough to have an application to treat this huge amount of data but must be converted to information, and this process is very expensive. Big data require a different and changing environment, which makes the process complicated. Providing necessary resources and expertise that many companies do not have the strategy should be formulated because the owner big data do not know what to do with them, where most of the companies are suffering from this gap; also, data supervisors should be employed, keep them safe and correct, prevent major problems [10]. Inappropriate diagnostics; there are people who use the Google search engine (big data) and do not always match the symptoms with the diagnosis. The difficulty of formulating public policy perceptions; viewing the information derived from social media is not an effective way because it does not differentiate between what is real and what is false. Increasing the burden on the systems; most administrations are not qualified to manage this amount of information and can also...
eBook - ePubBig Data Challenges
Society, Security, Innovation and Ethics
- Anno Bunnik, Anthony Cawley, Michael Mulqueen, Andrej Zwitter, Anno Bunnik, Anthony Cawley, Michael Mulqueen, Andrej Zwitter(Authors)
- 2016(Publication Date)
- Palgrave Macmillan(Publisher)
...© The Editor(s) (if applicable) and The Author(s) 2016 Anno Bunnik, Anthony Cawley, Michael Mulqueen and Andrej Zwitter (eds.) Big Data Challenges 10.1057/978-1-349-94885-7_1 Begin Abstract 1. Introduction to Big Data Challenges Anno Bunnik 1, Anthony Cawley 1, Michael Mulqueen 1 and Andrej Zwitter 1 (1) Department of Media and Communication, Liverpool Hope University, Liverpool, UK Abstract This chapter introduces the rationale for the book. It explains why Big Data is one of the most prominent challenges of our time, with far-reaching implications for society and security. It sets out why the tension, and interaction, between innovation and ethics is at the forefront of the various challenges of Big Data. It clarifies the division of the book into Part I, ‘Between Mathematics and Philosophy’, and Part II on ‘Implications for Security’. Each chapter is also briefly introduced to the reader. Keywords Introduction Big Data Security Society Innovation Ethics End Abstract This book unravels implications for society and security of a seismic shift in science, engineering, and computing. Driving the shift is ‘Big Data’, a term that loosely refers to remarkable advances in computational processing power, information storage, and clever software programmes. Whilst there is no universal definition of Big Data, it can be understood as referring to vast digital data sets that often capture very personal information of citizens and consumers. Examples would include Google searches, WhatsApp messages, or financial transaction flows by Walmart’s clientele. These data sets are increasingly recognised as a source to be harvested, aggregated, and analysed by, first and foremost, the private sector—corporate powerhouses have emerged in recent years that have adopted a business model largely on the promise of Big Data...
eBook - ePubBehavioral Competencies of Digital Professionals
Understanding the Role of Emotional Intelligence
- Sara Bonesso, Elena Bruni, Fabrizio Gerli(Authors)
- 2019(Publication Date)
- Palgrave Pivot(Publisher)
...Some examples are consumer rankings and reviews or the timeline of social media sites. self-quantification data, provided by individuals through personal actions and behaviors, such as that collected through wristbands that monitor fitness activities. The data can be uploaded in mobile device applications, tracked, and aggregated. To unlock the potential of this high volume of fast-moving and diverse data, technologies and analytics methods have made a leap forward in recent years, on the one hand, to capture, store, integrate, transform, and retrieve data (data management) and, on the other hand, to select the right model for analysis and to provide interpretations of the results (data analysis). 1.2 Data Science: How to Extract Value from Big Data The scientific body of knowledge that provides methods, processes, and systems to extract insights from data is defined as data science. It is an interdisciplinary field that combines statistics, computer science, data mining, machine learning, and analytics to understand and explain how we can generate analytical insights and prediction models from structured and unstructured big data. Ten years before the rise of the big data phenomenon, data science was defined by Cleveland as “an action plan to enlarge the technical areas of statistics” (Cleveland 2001 : 21). The advent of big data and its related challenges in data management and analysis have progressively expanded the domain of data science beyond the statistics field, assuming an increasing relevance. Figure 1.1 shows a Google Trends chart that displays web searches for the term “data science,” highlighting the dramatic increase in interest in data science in correspondence with the interest in big data. Fig. 1.1 Google trends for the keyword “data science” (July 2019) As the size of data (volume) is increasing at an exponential rate, scalability represents a key aim for models and new technologies that allow the storing and processing of a growing amount of data...
eBook - ePubHandbook of Research for Big Data
Concepts and Techniques
- Brojo Kishore Mishra, Vivek Kumar, Sanjaya Kumar Panda, Prayag Tiwari(Authors)
- 2022(Publication Date)
- Apple Academic Press(Publisher)
...This brings us to the challenges of data management, processing, and inference. The field of big data deals with data sets of large sizes or high complexity: managing such data sets, analyzing the data, extracting usable information from it, which shall not be in scope of or shall be non scalable for traditional data processing schemes. Big data is associated with five key concepts: volume, variety, velocity, veracity, and value. The ‘5 Vs’ characterize the architecture and working of the big data analytical scheme, laying the platform for extraction of vital information from given data sets [ 1 ]. The most important step towards data processing is the modeling of data. Data modeling is the process of realization of descriptive relational scheme so as to map the database towards various types of information that it shall constitute. The ability to systematically group key data points to be stored and retrieved, defining the relational function for its grouping comprises the data-modeling component of data science, serving the similar functionalities for complex datasets and hence big data science [ 1 ]. Considerations and challenges that affect data modeling and management schemes include: data ingestion, data storage, data quality, data operations, data scalability, and security challenges. Specific big data modeling with specific approaches include vector space models, graph data models, among other schemes. A quick example is the preventive machinery maintenance. We use big data from sensors (temperature, humidity, pressure, and vibration readings for each machinery part that come every second) to train, test, and retrain an ML model. The role of the model is to identify hidden patterns that lead to machinery failure and check newly incoming data against the identified patterns...
eBook - ePubBig Data Analytics and Intelligence
A Perspective for Health Care
- Poonam Tanwar, Vishal Jain, Chuan-Ming Liu, Vishal Goyal, Poonam Tanwar, Vishal Jain, Chuan-Ming Liu, Vishal Goyal(Authors)
- 2020(Publication Date)
- Emerald Publishing Limited(Publisher)
...These challenges are faced mainly with regards to rights, privacy, and trust.Fig. 1 Various Applications of Big Data in Health Care.The application mainly dependent on the healthcare section takes to help technologies that solve the issues based on computer diagnosis systems. The important task in this section is to upgrade the performance of the system to execute the user required computing.Fig. 1represents the various applications of big data in healthcare industry. Most widely used areas in which these enormous data can be implemented are mostly EHRs, to improvise security and privacy of the patients, patient predication, medical imaging, patient engagement, and telemedicine. These are the few areas where big data and analytics are used.2. Big Data OverviewThe term big data describes a huge amount of information which can be in any raw format are extracted from various sources in its raw format without making any changes. The users require a computing system that can be powerful enough to organize it and manipulate these data according to the needs of the user. Few examples of these sources are mobile, internet, social media, etc. Later stages of these raw data are used for further processing and can be used to make strategic decisions. With the help of big data and its analytical tools it enables by providing useful decisions that can be taken for future references. It also helps in understand data that were collected decades ago find solutions accordingly. Usually any problems related to data are solved by understanding its scope and impact.The data collected from big data can be mainly of three types or format. The three types are Volume, Velocity, and Variety. Volume mainly extracts the information from various channels, which includes websites that helps in establishing mass communication and interaction and information generated based on machine-to-machine processing. In the early stages of big data, the main issue was related to storage...
