Quantifying the User Experience

Practical Statistics for User Research

Jeff Sauro, James R Lewis

• Publisher: Morgan Kaufmann
• Year: 2016
• ISBN:
  9780128025482
• Edition: 2
• Topic: Computer Science
• Subtopic: Human-Computer Interaction
• Index: Computer Science

Quantifying the User Experience: Practical Statistics for User Research, Second Edition, provides practitioners and researchers with the information they need to confidently quantify, qualify, and justify their data. The book presents a practical guide on how to use statistics to solve common quantitative problems that arise in user research. It addresses questions users face every day, including, Is the current product more usable than our competition? Can we be sure at least 70% of users can complete the task on their first attempt? How long will it take users to purchase products on the website?

This book provides a foundation for statistical theories and the best practices needed to apply them. The authors draw on decades of statistical literature from human factors, industrial engineering, and psychology, as well as their own published research, providing both concrete solutions (Excel formulas and links to their own web-calculators), along with an engaging discussion on the statistical reasons why tests work and how to effectively communicate results. Throughout this new edition, users will find updates on standardized usability questionnaires, a new chapter on general linear modeling (correlation, regression, and analysis of variance), with updated examples and case studies throughout.

• Completely updated to provide practical guidance on solving usability testing problems with statistics for any project, including those using Six Sigma practices
• Includes new and revised information on standardized usability questionnaires
• Includes a completely new chapter introducing correlation, regression, and analysis of variance
• Shows practitioners which test to use, why they work, and best practices for application, along with easy-to-use Excel formulas and web-calculators for analyzing data
• Recommends ways for researchers and practitioners to communicate results to stakeholders in plain English

Chapter 1

Introduction and how to use this book

Abstract

The primary purpose of this book is to provide a statistical resource for those who measure the behavior and attitudes of people as they interact with interfaces. Our focus is on methods applicable to practical user research, based on our experience, investigations, and reviews of the latest statistical literature. As an aid to the persistent problem of remembering what method to use under what circumstances, this chapter contains four decision maps to guide researchers to the appropriate method and its chapter in this book.

Keywords

user experience

user research

summarizing data

margin of error

confidence interval

statistical test

comparison of groups

comparison with a benchmark

discrete-binary data

continuous data

decision map

Introduction

The last thing many designers and researchers in the field of user experience think of is statistics. In fact, we know many practitioners who find the field appealing because it largely avoids those impersonal numbers. The thinking goes that if usability and design are qualitative activities, it’s safe to skip the formulas and numbers.

Although design and several usability activities are certainly qualitative, the impact of good and bad designs can easily be quantified in conversions, completion rates, completion times, perceived satisfaction, recommendations, and sales. Increasingly, usability practitioners and user researchers are expected to quantify the benefits of their efforts. If they don’t, someone else will—unfortunately that someone else might not use the right metrics or methods.

The organization of this book

This book is intended for those who measure the behavior and attitudes of people as they interact with interfaces. This book is not about abstract mathematical theories for which you may someday find a partial use. Instead, this book is about working backwards from the most common questions and problems you’ll encounter as you conduct, analyze, and report on user research projects. In general, these activities fall into four areas:

1. Summarizing data and computing margins of error (Chapter 3)

2. Determining if there is a statistically significant difference, either in comparison to a benchmark (Chapter 4) or between groups (Chapters 5 and 10)

3. Finding the appropriate sample size for a study (Chapters 6, 7, and 10)

4. Investigating relationships among variables (Chapter 10).

We also provide:

• Background chapters with an overview of common ways to quantify user research (Chapter 2) and a quick introduction/review of many fundamental statistical concepts (Appendix)

• A comprehensive discussion of standardized usability questionnaires (Chapter 8)

• A discussion of enduring statistical controversies of which user researchers should be aware and able to articulate in defense of their analyses (Chapter 9)

• A wrap-up chapter with pointers to more information on statistics for user research (Chapter 11)

Each chapter ends with a list of key points and references. Most chapters also include a set of problems and answers to those problems so you can check your understanding of the content.

How to use this book

Despite there being a significant proportion of user research practitioners with advanced degrees, about 7% have PhDs (UXPA, 2014), for most people in the social sciences statistics is the only quantitative course they have to take. For many, statistics is a subject they know they should understand, but it often brings back bad memories of high-school math, poor teachers, and an abstract and difficult topic.

While we’d like to take all the pain out of learning and using statistics, there are still formulas, math, and some abstract concepts that we just can’t avoid. Some people want to see how the statistics work, and for them we provide the math. If you’re not terribly interested in the computational mechanics, then you can skip over the formulas and focus more on how to apply the procedures.

Readers who are familiar with many statistical procedures and formulas may find that some of the formulas we use differ from those taught in college statistics courses. Part of this is from recent advances in statistics (especially for dealing with binary data). Another part is due to our selecting the best procedures for practical user research, focusing on procedures that work well for the types of data and sample sizes you’ll likely encounter.

Based on teaching many courses at industry conferences and at companies, we know the statistics background of the readers of this book will vary substantially. Some o...