Python Social Media Analytics

Siddhartha Chatterjee, Michal Krystyanczuk

• Éditeur: Packt Publishing
• Année: 2017
• ISBN:
  9781787126756
• Édition: 1
• Sujet: Ciencia de la computación
• Sous-sujet: Minería de datos

Leverage the power of Python to collect, process, and mine deep insights from social media dataAbout This Book• Acquire data from various social media platforms such as Facebook, Twitter, YouTube, GitHub, and more• Analyze and extract actionable insights from your social data using various Python tools• A highly practical guide to conducting efficient social media analytics at scaleWho This Book Is ForIf you are a programmer or a data analyst familiar with the Python programming language and want to perform analyses of your social data to acquire valuable business insights, this book is for you. The book does not assume any prior knowledge of any data analysis tool or process.What You Will Learn• Understand the basics of social media mining• Use PyMongo to clean, store, and access data in MongoDB• Understand user reactions and emotion detection on Facebook• Perform Twitter sentiment analysis and entity recognition using Python• Analyze video and campaign performance on YouTube• Mine popular trends on GitHub and predict the next big technology• Extract conversational topics on public internet forums• Analyze user interests on Pinterest• Perform large-scale social media analytics on the cloudIn DetailSocial Media platforms such as Facebook, Twitter, Forums, Pinterest, and YouTube have become part of everyday life in a big way. However, these complex and noisy data streams pose a potent challenge to everyone when it comes to harnessing them properly and benefiting from them. This book will introduce you to the concept of social media analytics, and how you can leverage its capabilities to empower your business.Right from acquiring data from various social networking sources such as Twitter, Facebook, YouTube, Pinterest, and social forums, you will see how to clean data and make it ready for analytical operations using various Python APIs. This book explains how to structure the clean data obtained and store in MongoDB using PyMongo. You will also perform web scraping and visualize data using Scrappy and Beautifulsoup.Finally, you will be introduced to different techniques to perform analytics at scale for your social data on the cloud, using Python and Spark. By the end of this book, you will be able to utilize the power of Python to gain valuable insights from social media data and use them to enhance your business processes.Style and approachThis book follows a step-by-step approach to teach readers the concepts of social media analytics using the Python programming language. To explain various data analysis processes, real-world datasets are used wherever required.

Scraping and Extracting Conversational Topics on Internet Forums

In the last few chapters, we have explored several social networks to analyze the data that we gathered through their APIs. However, there is another world, the world of internet forums or message boards, which most often do not have APIs. Even though they are public, internet forums differ significantly from social networks, as their users are mostly anonymous. This is unlike social networks, where, more often than not, users make their identities available. The anonymous nature of these platforms leads to free and fearless topical discussions on almost every subject on Earth. Technology, health, religion, politics, social activism, markets and industries, and movies are just a few of the threads or topics on which millions of people engage in deep discussions on online forums. Unlike Twitter, which has a limitation on the number of characters on each tweet, forums have no such limits. Therefore, online forums are a better alternate source of information than social networks about people and for companies and their consumers. Many companies are analyzing discussions on forums about their products, categories, and promotions to gather invaluable insights about their business. The lack of APIs to collect this conversation data means that we need to create crawlers or spiders that can parse the data from the pages of the forums and structure the conversational data. The art of crawling internet data could be used for almost every content on web pages that doesn't prohibit us from doing so. Google and other search engines crawl billions of web pages on the Internet to effectively rank them on search engine results. We shall use similar techniques using Python modules to gather data from online forums. Gathering data from online forums is one task, but making sense out of it is quite another.

Since the textual content from forums is rich and topical, we need sophisticated techniques to understand them. We shall use a generative statistical modeling technique called a Latent Dirichlet Algorithm, part of a family of algorithms known as Topic Modeling, to extract topics from the textual data. This is by far one of the most powerful text mining techniques that can be used by data scientists.

Scope and process

The process of topic modelling that we will present in this chapter consists of three steps:

1. Data crawling.
2. Data pre-processing.
3. Topic modelling and interpretation.

Each step represents a logical data analysis flow, which will finally lead us to different clusters of topics.

First, we will try to find out what the main topics are:

Getting the data

Forums do not provide programmatic interfaces (APIs) to capture data. However, you can connect to the website as a user to see all the conversations and collect data. The process of data extraction automatically from websites is called 'web scraping'.

Introduction to scraping

Since the beginning of the web, web scraping has been the main challenge for anyone who wanted to exploit the richness of information available on the Internet. In the very beginning, very few APIs were available and people used to copy the content of websites by just using copy-paste schema. Then, some programmatic tools were created to follow links (crawling) and extract the content from web pages (scraping). The information was structured by using text patterns (regex) or DOM (Document Object Model) parsing methods. More recently, the development of semantic analysis tools and artificial intelligence enabled alternative approaches, which are much more efficient and closer to human understanding and interpretation of website content. Search engines, especially Google, have been leaders in web scraping, as they go about crawling the entire web to index content from web pages and make it available through its search engine for the entire world. However, nowad...