Python Automation Cookbook

75 Python automation ideas for web scraping, data wrangling, and processing Excel, reports, emails, and more, 2nd Edition

Jaime Buelta

• Publisher: Packt Publishing
• Year: 2020
• eBook ISBN: 9781800202597
• Edition: 2
• Topic: Computer Science
• Subtopic: Data Processing
• Index: Computer Science

Get a firm grip on the core processes including browser automation, web scraping, Word, Excel, and GUI automation with Python 3.8 and higher

Key Features

• Automate integral business processes such as report generation, email marketing, and lead generation
• Explore automated code testing and Python's growth in data science and AI automation in three new chapters
• Understand techniques to extract information and generate appealing graphs, and reports with Matplotlib

Book Description

In this updated and extended version of Python Automation Cookbook, each chapter now comprises the newest recipes and is revised to align with Python 3.8 and higher. The book includes three new chapters that focus on using Python for test automation, machine learning projects, and for working with messy data.

This edition will enable you to develop a sharp understanding of the fundamentals required to automate business processes through real-world tasks, such as developing your first web scraping application, analyzing information to generate spreadsheet reports with graphs, and communicating with automatically generated emails.

Once you grasp the basics, you will acquire the practical knowledge to create stunning graphs and charts using Matplotlib, generate rich graphics with relevant information, automate marketing campaigns, build machine learning projects, and execute debugging techniques.

By the end of this book, you will be proficient in identifying monotonous tasks and resolving process inefficiencies to produce superior and reliable systems.

What you will learn

• Learn data wrangling with Python and Pandas for your data science and AI projects
• Automate tasks such as text classification, email filtering, and web scraping with Python
• Use Matplotlib to generate a variety of stunning graphs, charts, and maps
• Automate a range of report generation tasks, from sending SMS and email campaigns to creating templates, adding images in Word, and even encrypting PDFs
• Master web scraping and web crawling of popular file formats and directories with tools like Beautiful Soup
• Build cool projects such as a Telegram bot for your marketing campaign, a reader from a news RSS feed, and a machine learning model to classify emails to the correct department based on their content
• Create fire-and-forget automation tasks by writing cron jobs, log files, and regexes with Python scripting

Who this book is for

Python Automation Cookbook - Second Edition is for developers, data enthusiasts or anyone who wants to automate monotonous manual tasks related to business processes such as finance, sales, and HR, among others. Working knowledge of Python is all you need to get started with this book.

12

Automatic Testing Routines

In this chapter, we will cover the following recipes:

• Writing and executing test cases
• Testing external code
• Testing using dependency mocking
• Testing using HTTP call mocking
• Preparing testing scenarios
• Running tests selectively

Introduction

When the code and the complexity of your software grows, generating tests to ensure your program does what it is supposed to do is the best tool to provide you with a firm footing over rocky terrain.

Tests are, in essence, double-checking the code is valid and doing what it is supposed to do. This is a deceptively simple statement, but in practice, it can be very difficult.

Mastering the ability to test is a difficult task and is worth a book or two. The tasks introduced in this chapter move from business-oriented tasks to software engineering tasks, which is a different approach. The objective of this chapter is to present some practical aspects of testing to introduce the subject.

The most important thing about a test is that it tries to check the code that is being tested independently. This involves a mindset of looking at the code and the inputs and outputs and approaching the task with a fresh look, and to not be influenced by the internal implementation. In some cases, the people testing the software may be different from the people writing the code in the first place, to ensure that there's a good understanding of what the code should and should not do. Try to design your tests with this approach and create well-defined interfaces to work with.

In summary, testing checks the code does what it is supposed to do, and doesn't do what it is not supposed to do.

The first part of the sentence is easier, but the second part is very difficult or even impossible. Writing tests has a cost, both in time and in maintenance. Only in highly critical software is code tested extensively to ensure that nothing unexpected happens. Try to find your balance in how many tests are adequate for your needs.

Tests are typically classified depending on how many different parts of the software they test. A test that only covers a small part of the code, like a function, is normally called a unit test, while a test that covers the whole system is called a system test. A test that covers the integration of different software elements is called an integration test. This is a very fluid definition, as not everyone agrees on what the system is, or how big a unit test can be before it becomes an integration test, or whether there's a significant difference between integration and system tests. But it helps to understand that different tests cover different areas of software – some are bigger, some are smaller – and have different requirements.

To have a team of developers working on the same software, ensuring it is of high quality, and the discipline of adding and running tests continuously is crucial. That's why there are lots of types of tests to help with that task. Continuous integration, or CI, includes the practice of running tests for each change to the software and before merging them with changes by other developers. CI tools allow you to run tests in the background automatically and notify developers of any problem, ensuring that there are no unexpected failures.

In this chapter, we will cover the usage of pytest as a tool to run various tests. This is one of the most complete Python test frameworks. pytest enables the easy setup of tests and provides a lot of useful options to run subsets of tests, run tests quickly, and detect problems when tests fail.

It has also an extensive ecosystem of plugins that allows you to integrate with other systems, like databases and web services, and to extend pytest with additional features, such as code coverage, running tests in parallel, and performance benchmarking. Let's start writing and running some simple tests.

Writing and executing test cases

In this recipe, we will learn how to define and run tests using the pytest library.

Getting ready

We will use the pytest module. We should install the module, adding it to our requirements.txt file as follows:

$ echo "pytest==5.4.1" >> requirements.txt $ pip install -r requirements.txt 

We will use the file test_case.py. You can download it from the GitHub repository at https://github.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/tree/master/Chapter12/tests.

How to do it...

1. Check the file tests/test_case.py, which contains the definition of four tests:

   LIST = [1, 2, 3] def test_one(): # Nothing happens pass def test_two(): assert 2 == 1 + 1 def test_three(): assert 3 in LIST def test_fail(): assert 4 in LIST 

2. Run pytest to run the test file:

   $ pytest tests/test_case.py =====================. test session starts ====================== platform darwin -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 rootdir: /Python-Automation-Cookbook/Chapter12 collected 4 items tests/test_case.py ...F [100%] ============================= FAILURES =========================== _____________________________ test_fail __________________________ de...

Table of contents

• Preface
• Let's Begin Our Automation Journey
• Automating Tasks Made Easy
• Building Your First Web Scraping Application
• Searching and Reading Local Files
• Generating Fantastic Reports
• Fun with Spreadsheets
• Cleaning and Processing Data
• Developing Stunning Graphs
• Dealing with Communication Channels
• Why Not Automate Your Marketing Campaign?
• Machine Learning for Automation
• Automatic Testing Routines
• Debugging Techniques
• Other Books You May Enjoy
• Index