Pandas 1.x Cookbook

Practical recipes for scientific computing, time series analysis, and exploratory data analysis using Python, 2nd Edition

Matt Harrison, Theodore Petrou

• Publisher: Packt Publishing
• Year: 2020
• ISBN:
  9781839218910
• Edition: 2
• Topic: Computer Science
• Subtopic: Data Processing
• Index: Computer Science

Use the power of pandas to solve most complex scientific computing problems with ease. Revised for pandas 1.x.

Key Features

• This is the first book on pandas 1.x
• Practical, easy to implement recipes for quick solutions to common problems in data using pandas
• Master the fundamentals of pandas to quickly begin exploring any dataset

Book Description

The pandas library is massive, and it's common for frequent users to be unaware of many of its more impressive features. The official pandas documentation, while thorough, does not contain many useful examples of how to piece together multiple commands as one would do during an actual analysis. This book guides you, as if you were looking over the shoulder of an expert, through situations that you are highly likely to encounter.

This new updated and revised edition provides you with unique, idiomatic, and fun recipes for both fundamental and advanced data manipulation tasks with pandas. Some recipes focus on achieving a deeper understanding of basic principles, or comparing and contrasting two similar operations. Other recipes will dive deep into a particular dataset, uncovering new and unexpected insights along the way. Many advanced recipes combine several different features across the pandas library to generate results.

What you will learn

• Master data exploration in pandas through dozens of practice problems
• Group, aggregate, transform, reshape, and filter data
• Merge data from different sources through pandas SQL-like operations
• Create visualizations via pandas hooks to matplotlib and seaborn
• Use pandas, time series functionality to perform powerful analyses
• Import, clean, and prepare real-world datasets for machine learning
• Create workflows for processing big data that doesn't fit in memory

Who this book is for

This book is for Python developers, data scientists, engineers, and analysts. Pandas is the ideal tool for manipulating structured data with Python and this book provides ample instruction and examples. Not only does it cover the basics required to be proficient, but it goes into the details of idiomatic pandas.

12

Time Series Analysis

Introduction

The roots of pandas lay in analyzing financial time series data. Time series are points of data gathered over time. Generally, the time is evenly spaced between each data point. However, there may be gaps in the observations. pandas includes functionality to manipulate dates, aggregate over different time periods, sample different periods of time, and more.

Understanding the difference between Python and pandas date tools

Before we get to pandas, it can help to be aware of and understand core Python's date and time functionality. The datetime module provides three data types: date, time, and datetime. Formally, a date is a moment in time consisting of just the year, month, and day. For instance, June 7, 2013 would be a date. A time consists of hours, minutes, seconds, and microseconds (one-millionth of a second) and is unattached to any date. An example of time would be 12 hours and 30 minutes. A datetime consists of both the elements of a date and time together.

On the other hand, pandas has a single object to encapsulate date and time called a Timestamp. It has nanosecond (one-billionth of a second) precision and is derived from NumPy's datetime64 data type. Both Python and pandas each have a timedelta object that is useful when doing date addition and subtraction.

In this recipe, we will first explore Python's datetime module and then turn to the corresponding date tools in pandas.

How to do it…

1. Let's begin by importing the datetime module into our namespace and creating a date, time, and datetime object:

   >>> import pandas as pd >>> import numpy as np >>> import datetime >>> date = datetime.date(year=2013, month=6, day=7) >>> time = datetime.time(hour=12, minute=30, ... second=19, microsecond=463198) >>> dt = datetime.datetime(year=2013, month=6, day=7, ... hour=12, minute=30, second=19, ... microsecond=463198) >>> print(f"date is {date}") date is 2013-06-07 >>> print(f"time is {time}") time is 12:30:19.463198 >>> print(f"datetime is {dt}") datetime is 2013-06-07 12:30:19.463198 

2. Let's construct and print out a timedelta object, the other major data type from the datetime module:

   >>> td = datetime.timedelta(weeks=2, days=5, hours=10, ... minutes=20, seconds=6.73, ... milliseconds=99, microseconds=8) >>> td datetime.timedelta(days=19, seconds=37206, microseconds=829008) 

3. Add this td to the date and dt objects from step 1:

   >>> print(f'new date is {date+td}') new date is 2013-06-26 >>> print(f'new datetime is {dt+td}') new datetime is 2013-06-26 2...