Many think that the big step from "coding" to "software engineering" is made by having elegant architectures, well-defined execution plans, and software that moves big companies' processes. This mostly comes from our vision of the classic industrial product development world, where planning mostly mattered more than execution, because the execution was moved forward by an assembly line and software was an expensive internal utility that only big companies could afford
As software development science moved forward and matured, it became clear that classic industrial best practices weren't always a great fit for it. The reason being that every software product was very different, due to the technologies involved, the speed at which those technologies evolve, and in the end the fact that different software had to do totally different things. Thus the idea developed that software development was more similar to craftsmanship than to industry.
If you embrace that it's very hard, and not very effective, to try to eliminate uncertainty and issues with tons of preparation work due to the very nature of software itself, it becomes evident that the most important part of software development is detecting defects and ensuring it achieves the expected goals. Those two things are usually mostly done by having tests and a fitness function that can verify the software does what we really mean it to – founding pieces of the whole Software Quality Control discipline, which is what this chapter will introduce and, in practice, what this book is all about.
In this chapter, we will go through testing software products and the best practices in quality control. We will also introduce automatic tests and how they are superseding manual testing. We will take a look at what Test-Driven Development (TDD) is and how to apply it in Python, giving some guidance on how to distinguish between the various categories of tests, how to implement them, and how to get the right balance between test efficacy and test cost.
In this chapter, we will cover the following:
- Introducing software testing and quality control
- Introducing automatic tests and test suites
- Introducing test-driven development and unit tests
- Understanding integration and functional tests
- Understanding the testing pyramid and trophy
Technical requirements
A working Python interpreter is all that's needed.
The examples have been written in Python 3.7 but should work in most modern Python versions.
You can find the code files present in this chapter on GitHub at https://github.com/PacktPublishing/Crafting-Test-Driven-Software-with-Python/tree/main/Chapter01.
Introducing software testing and quality control
From the early days, it was clear that like any other machine, software needed a way to verify it was working properly and was built with no defects.
Software development processes have been heavily inspired by manufacturing industry standards, and early on, testing and quality control were introduced into the product development life cycle. So software companies frequently have a quality assurance team that focuses on setting up processes to guarantee robust software and track results.
Those processes usually include a quality control process where the quality of the built artifact is assessed before it can be considered ready for users.
The quality control process usually achieves such confidence through the execution of a test plan. This is usually a checklist that a dedicated team goes through during the various phases of production to ensure the software behaves as expected.
Test plans
A test plan is composed of multiple test cases, each specifying the following:
- Preconditions: What's necessary to be able to verify the case
- Steps: Actions that have to succeed when executed in the specified order
- Postconditions: In which state the system is expected to be at the end of the steps
A sample test case of software where logging in with a username and password is involved, and we might want to allow the user to reset those, might look like the following table:
| Test Case: 2.2 - Change User Password |
| Preconditions: - A user, user1 exists
- The user is logged in as user1
- The user is at the main menu
|
| # | Action | Expected Response | Success / Fail | | 1 | Click the change password button. | The system shows a dialog to insert a new password. | | | 2 | Enter newpass. | The dialog shows 7 asterisks in the password field. | | | 3 | Click the OK button. | The system shows a dialog with a success message. | | | 4 | Wait 2 seconds. | The success dialog goes away. | | |
| Postconditions: - The user1 password is now newpass
|
These test cases are divided into cases, are manually verified by a dedicated team, and a sample of them is usually selected to be executed during development, but most of them are checked when the development team declared the work done.
This meant that once the team finishes its work, it takes days/weeks for the release to happen, as the whole software has to be verified by humans clicking buttons, with all the unpredictable results that involves, as humans can get distracted, pressing the wrong button or receiving phone calls in the middle of a test case.
As software usage became more widespread, and business-to-consumer products became the norm, consumers started to appreciate faster release cycles. Companies that updated their products with new features frequently were those that ended up dominating the market in the long term.
If you think about modern release cycles, we are now used to getting a new version of our favorite mobile application weekly. Such applications are probably so complex that they involve thousands of test cases. If all those cases had to be performed by a human, there would be no way for the company to provide you with frequent releases.
The worst thing you can do, by the way, is to release a broken product. Your users will lose confidence and will switch to other more reliable competitors if they can't get their job done due to crashes or bugs. So how can we deliver such frequent releases without reducing our test coverage and thus incurring more bugs?
The solution came from automating the test process. So while we learned how to detect defects by writing and executing test plans, it's only by making them automatic that we can scale them to the number of cases that will ensure robust software in the long term.
Instead of having humans test software, have some other software test it. What a person does in seconds can happen in milliseconds with software and you can run thousands of tests in a few minutes.
Introducing automatic tests and test suites
Automated testing is, in practice, the art of writing another piece of software to test an original piece of software.
As testing a whole piece of software has to take millions of variables and possible code paths into account, a single program trying to test another one would be very complex and hard to maintain. For this reason, it's usually convenient to split that program into smaller isolated programs, each being a test case.
Each test case contains all the instructions that are required to set up the target software in a state where the parts that are the test case areas of interest can be tested, the tests can be done, and all the conditions can be verified and reset back to the state of the target software so a subsequent test case can find a known state from which to start.
When using the unittest module that comes with the Python Standard Library, each test case is declared by subclassing from the unittest.TestCase class and adding a method whose name starts with test, which will contain the test itself:
import unittest
class MyTestCase(unittest.TestCase):
def test_one(self):
pass
Trying to run our previous test will do nothing by the way:
$ python 01_automatictests.py
$
We declared our test case, but we have nothing that runs it.
As for manually executed tests, the automatic tests need someone in charge of gathering all test cases and running the...
