A Python Guide for Web Scraping

Explore Python Tools, Web Scraping Techniques, and How to Automata Data for Industrial Applications (English Edition)

Pradumna Milind Panditrao

• Publisher: BPB Publications
• Year: 2021
• ISBN:
  9789390684991
• Topic: Informatik
• Subtopic: Natürliche Sprachverarbeitung

Get hands-on training on any web crawling/scraping tool and uses of web scraping in the real-time industry

Key Features
? Includes numerous use-cases on the use of web scraping for industrial applications.
? Learn how to automate web scraping tasks.
? Explore ready-made syntaxes of Python scripts to run web scraping.

Description
A Python Guide for Web Scraping is a book that will give information about the importance of web scraping using Python. It includes real-time examples of web scraping. It implies the automation use cases of web scraping as well. It gives information about the different tools and libraries of web scraping so that readers get a wide idea about the features and existence of web scraping.In this book, we started with the basics of Python and its syntactical information. We briefed about the use cases and features of Python. We have explained the importance of Python in automation systems. Furthermore, we have added information about real-time industrial examples. We have concentrated and deep-dived into Python's importance in web scraping, explained the different tools and their usages. We have explained the real-time industrial domain-wise use cases for web scraping.

What you will learn
? Explore the Python syntax and key features of using Python for web scraping.
? Usage of Python in the web scraping tasks and how to automate scraping.
? How to use different libraries and modules of Python.

Who this book is for
This book is basically for data engineers and data programmers who have a basic knowledge of Python and for the readers who want to learn about web scraping projects for industries.

Table of Contents
1. Python Basics
2. Use Cases of Python
3. Automation Using Python
4. Industrial Automation-Python
5. Web Scraping
6. Web Scraping and Necessity
7. Python - Web Scraping and Different Tools
8. Automation in Web Scraping
9. Use Cases-Web Scraping
10. Industrial Benefits of Web Scraping

About the Authors
Mr Pradumna Panditrao is currently working as a Senior Software Engineer and a DevOps tool developer. He has done his Masters in networking and telecommunications. He has a total of 8+ years of experience in various domains like Software Development, DevOps Automation tools, Data mining Crawling tools, Cloud Technologies, and Hardware Profiling. He has good exposure to the cloud and has published a paper on Cognitive Radio, 4G Technology Algorithms. He has given embedded software development lectures and lab demo sessions at Bits Pilani, Goa in 2014-2015.

CHAPTER 1

Python Basics

1.1 Introduction

Python is one of the most famous programming languages in software development. Python is an interpreted, high-level, general-purpose programming language. It is a general-purpose coding language—which means that unlike HTML, CSS, and JavaScript, it can be used for programming and software development in addition to web development.

1.2 Structure

In this chapter we will discuss the following topics:

• History of Python
• Basics of Python
• Python internals
• Installing Python on different operating systems
• Different aspects of Python
• Capabilities of Python

1.3 Objective

After studying this chapter, you should be able to:

• Understand the Python history
• Install Python on different operating systems
• Understand Python from different aspects and its capabilities

1.4 Python and its history

Python is a general-purpose, high-level, interpreted, object-oriented open source programming language. Python was designed in the 1980s by Guido van Rossum as a side project and later implemented by the Python Software Foundation (PSF). Python was finally released in 1991. It was designed to reduce the large code size of C, JAVA, and C++, in turn improving time efficiency and productivity of developers.

Nowadays Python is famous for different reasons. Python has compatibility benefits with different versions of it. Python can be used for different purposes such as web development (GUI applications, websites), system-level programming, machine-level programming, developing IoT applications, etc. As Python is highly abstracted from the assembly language, it can be used to provide instructions to the CPU. It is helpful for rapid application programming as it has a high-level build in data structures. Python offers fast development cycle with test cycle speeds as it is not affected by segmentation fault errors, making debugging very easy. As it is an interpreted language, in case of errors, it returns an exception. When the program doesn't catch the exception, the interpreter prints a stack trace.

1.4.1 Python versions

Python 3.7.x and 3.9.0 are the latest versions. The two most widely used versions of Python are Python 2.x and 3.x. There is a lot of competition between the two and both of them seem to have quite a number of different supporters. As of January 1, 2020, the 2.x branch of the Python programming language is no longer supported by its developers, the Python Software Foundation.

Python is used for various purposes such as developing, scripting, generation, and software testing. Due to its beauty for programming and simplicity, top organizations like, Google, Quora, Mozilla, Hewlett-Packard, Qualcomm, IBM, and Cisco have implemented Python for development.

Figure 1.1: Python version Cycle

1.5 Preference of Python over others

Following are the reasons for choosing Python as your programming language:

• Open-source framework: Basically, the Python language is open-source, so we can use several open-source Python frameworks, libraries, and development tools to reduce development time and operating costs. Programming is easy with an IDE.
• Useful and robust libraries: Python offers a large collection of useful libraries which can support all possible use cases of a developer. Different modules can be used by developers to meet their specific needs.
• Compatibility with different platforms and systems: Python is supported on multiple operating systems like OSx, Windows, Linux, etc. You can build Python on specific versions, get dependencies, and make compatible versions. For example, you can create a Python program on CentOS 4.5 and run it on a later version of the operating system.
• Supports prototype-based programming: It supports different prototypes like object-orientation support. It supports automatic memory management and multi-threaded programming. This feature is useful for designing very complex applications with a variety of features.

  

  Figure 1.2: Python usability
• Easy to maintain: Maintenance of application and reusability of code are the top-most priorities of any programmer. Python definitely fits in these criteria as it supports virtual environments (we will cover virtual environments later in the chapter). It means that you can change the version of Python as per the requirement of your application.
• Readable: Python code has high-readability. We can customize a Python application without adding much code due its clean code base.
• Test cycle driven: As application development is rapid with Python, its testing cycle is also fast. Python can be used to design APIs based on ...