Measuring the User Experience

Collecting, Analyzing, and Presenting Usability Metrics

Bill Albert, Tom Tullis

• Publisher: Morgan Kaufmann
• Year: 2013
• ISBN:
  9780124157927
• Edition: 2
• Topic: Informatique
• Subtopic: Interaction homme-machine

Measuring the User Experience was the first book that focused on how to quantify the user experience. Now in the second edition, the authors include new material on how recent technologies have made it easier and more effective to collect a broader range of data about the user experience.

As more UX and web professionals need to justify their design decisions with solid, reliable data, Measuring the User Experience provides the quantitative analysis training that these professionals need. The second edition presents new metrics such as emotional engagement, personas, keystroke analysis, and net promoter score. It also examines how new technologies coming from neuro-marketing and online market research can refine user experience measurement, helping usability and user experience practitioners make business cases to stakeholders. The book also contains new research and updated examples, including tips on writing online survey questions, six new case studies, and examples using the most recent version of Excel.

• Learn which metrics to select for every case, including behavioral, physiological, emotional, aesthetic, gestural, verbal, and physical, as well as more specialized metrics such as eye-tracking and clickstream data
• Find a vendor-neutral examination of how to measure the user experience with web sites, digital products, and virtually any other type of product or system
• Discover in-depth global case studies showing how organizations have successfully used metrics and the information they revealed
• Companion site, www.measuringux.com, includes articles, tools, spreadsheets, presentations, and other resources to help you effectively measure the user experience

Chapter 1

Introduction

Contents

1.1 What is User Experience

1.2 What are User Experience Metrics?

1.3 The Value of UX Metrics

1.4 Metrics for Everyone

1.5 New Technologies in UX Metrics

1.6 Ten Myths about UX Metrics

Myth 1: Metrics Take Too Much Time to Collect

Myth 2: UX Metrics Cost Too Much Money

Myth 3: UX Metrics are not Useful When Focusing on Small Improvements

Myth 4: UX Metrics Don’t Help us Understand Causes

Myth 5: UX Metrics are Too Noisy

Myth 6: You Can Just Trust Your Gut

Myth 7: Metrics Don’t Apply to New Products

Myth 8: No Metrics Exist for the Type of Issues We are Dealing with

Myth 9: Metrics are not Understood or Appreciated by Management

Myth 10: It’s Difficult to Collect Reliable Data with a Small Sample Size

The goal of this book is to show how user experience (UX) metrics can be a powerful tool for evaluating and improving the design of any product. When some people think about user experience metrics, they feel overwhelmed by complicated formulas, contradictory research, and advanced statistical methods. We hope to demystify much of the research and focus on the practical application of UX metrics. We’ll walk you through a step-by-step approach to collecting, analyzing, and presenting UX metrics. We’ll help you choose the right metrics for each situation or application and show you how to use them to produce reliable, actionable results without breaking your budget. We will introduce some new UX metrics that you might want to consider adding to your toolkit. We’ll give you guidelines and tips for analyzing a wide range of metrics and provide many different examples of how to present UX metrics to others in simple and effective ways.

Our intention is to make this book a practical, how-to guide about measuring the user experience of any product. We aren’t going to give you a lot of formulas; in fact, there are very few. The statistics are fairly limited, and the calculations can be done easily in Excel or some other common software package or web application. Our intention is to give you the tools you need to evaluate the user experience of nearly any type of product, without overwhelming you with unnecessary details.

This book is both product and technology neutral. The UX metrics we describe can be used for practically any type of product utilizing nearly any type of technology. This is one of the great features of UX metrics: they aren’t just for websites or any single technology. For example, task success and satisfaction are equally valid whether you evaluate a website, a smartphone, or a microwave oven.

The “half-life” of UX metrics is much greater than any specific design or technology. Despite all the changes in technology, the metrics essentially stay the same. Some metrics may change with the development of new technologies to measure the user experience, but the underlying phenomena being measured don’t change. Eye tracking is a great example. Many researchers wanted a method for determining where exactly someone is looking at any point in time. Now, with the latest advances in eye-tracking technology, measurement has become much easier and far more accurate. The same can be said for measuring emotional engagement. New technologies in affective computing allow us to measure levels of arousal through very unobtrusive skin conductance monitors as well as facial recognition software. This has offered glimpses into the emotional state of users as they interact with different types of products. These new technologies for measurement are no doubt extremely useful; however, the underlying questions we are all trying to answer don’t change that much at all.

So why did we write this book? There’s certainly no shortage of books on human factors, statistics, experimental design, and usability methods. Some of those books even cover the more common UX metrics. Does a book that focuses entirely on UX metrics even make sense? Obviously, we think so. In our (humble) opinion, this book makes five unique contributions to the realm of user experience research:

• We take a comprehensive look at UX metrics. No other books review so many different metrics. We provide details on collecting, analyzing, and presenting a diverse range of UX metrics.

• This book takes a practical approach. We assume you’re interested in applying UX metrics as part of your job. We don’t waste your time with unnecessary details. We want you to be able to use these metrics easily every day.

• We provide help in making the right decisions about UX metrics. One of the most difficult aspects of a UX professional’s job is deciding whether to collect metrics and, if so, which ones to use. We guide you through the decision process so that you find the right metrics for your situation.

• We provide many examples of how UX metrics have been applied within different organizations and how they have been used to answer specific research questions. We also provide in-depth case studies to help you determine how best to use the information revealed by the UX metrics.

• We present UX metrics that can be used with many different products or technologies. We take a broad view so that these metrics can be helpful throughout your career even as technology evolves and products change.

This book is organized into three main parts. The first part (Chapters 1–3) provides background information needed to get up to speed on UX metrics.

• Chapter 1 provides an overview of user experience and metrics. We define user experience, discuss the value of measuring the user experience, share some of the emerging trends, dispel some of the common myths about UX metrics, and introduce some of the newest concepts in UX measurement.

• Chapter 2 includes background information on UX data and some basic statistical concepts. We also provide a guide for performing common statistical procedures related to different UX methods.

• Chapter 3 focuses on planning a study involving metrics, including defining participant goals and study goals and choosing the right metrics for a wide variety of situations.

The second part (Chapters 4–9) reviews five general types of UX metrics, as well as some special topics that don’t fall neatly into any single type. For each metric, we explain what it is, when to use it, and when not to use it. We show you how to collect data and different ways to analyze and present it. We provide examples of how it has been used in real-world user experience research.

• Chapter 4 covers various types of performance metrics, including task success, time on task, errors, efficiency, and ease of learning. These metrics are grouped under an “umbrella” of performance because they measure different aspects of the user’s behavior.

• Chapter 5 looks at measuring usability issues. Usability issues can be quantified easily by measuring the frequency, severity, and type of issue. We also discuss some of the debates about appropriate sample sizes and how to capture usability issues reliably.

• Chapter 6 focuses on self-reported metrics, such as satisfaction, expectations, ease-of-use ratings, confidence, usefulness, and awareness. Self-reported metrics are based on what users share about their experiences, not what the UX professional measures about their actual behaviors.

• Chapter 7 is devoted to behavioral and physiological metrics. These metrics include eye tracking, emotional engagement, facial expressions, and various measures of stress. All of these metrics capture something about how the body behaves as a result of the experience of interacting with a user interface.

• Chapter 8 discusses how to combine different types of metrics and derive new metrics. Sometimes it’s helpful to get an overall assessment of the user experience of any product. This global assessment is achieved by combining different types of metrics into a single UX score, summarizing them in a UX scorecard, or comparing them to expert performance.

• Chapter 9 presents special topics that we believe are important but that don’t fit squarely into one of the five general categories. These include A/B testing on a live website, card-sorting data, accessibility data, and return on investment (ROI).

The third part (Chapters 10 and 11) shows how UX metrics are put into practice. In this part, we highlight how UX metrics are actually used within different types of organizations and how to promote the use of metrics within an organization.

• Chapter 10 presents five case studies. Each case study reviews how different types of UX metrics were used, how data were collected and analyzed, and the results. These case studies were drawn from UX professionals in various types of organizations, including consulting, government, industry, and not-for-profit/education.

• Chapter 11 provides 10 steps to help you move forward in using metrics within your organization. We discuss how UX metrics can fit within different types of organizations, practical tips for making metrics work within your organization, and recipes for success.

1.1 What is User Experience

Before we try to measure user experience, we should know what it is and what it isn’t. While many UX professionals have their own ideas of what constitutes a “user experience,” we believe the user experience includes three main defining characteristics:

• A user is involved

• That user is interacting with a product, system, or really anything with an interface

• The users’ experience is of interest, and observable or measurable

In the absence of a user doing something, we might just be measuring attitudes and preferences, such as in a political poll or survey about your favorite flavor of ice cream. There has to be behavior, or at least potential behavior, to be considered user experience. For example, we might show a screenshot of a website and ask participants what they would click if it were interactive.

You might also note that we never defined any characteristics of the product or system. We believe that any system or product can be evaluated from a user experience perspective, as long as there is some type of interface between the system or product and the user. We are hard-pressed to think of any examples of a product that don’t have some type of human interface. We think that’s a good thing, as it means that we can study almost any product or system from a UX perspective.

Some people distinguish between the terms usability and user experience. Usability is usually considered the ability of the user to use the thing to carry out a task successfully, whereas user experience takes a broader view, looking at the individual’s entire interaction with the thing, as well as the thoughts, feelings, and perceptions that result from that interaction.

In any casual conversation about usability, most people would agree that it’s good to have something that works well and isn’t confusing to use. On the other side of the coin, some companies may intentionally design products to be confusing or frustrating. Fortunately, this is a rare occurrence. For the purposes of this book, we will be somewhat idealistic and make the assumption that both users and designers want products to be easy to use, efficient, and engaging.

User experience can sometimes mean the difference between life and death. For example, the health industry is not immune to poor usability. Usability issues abound in medical devices, procedures, and even diagnostic tools. Jakob Nielsen (2005) cites one study that found 22 separate usability issues that contributed to patients receiving the wrong medicine. Even more troubling is that, on average, 98,000 Americans die every year due to medical error (Kohn et al., 2000). While there are no doubt many factors behind this, some speculate that usability and human factors are at least partially to blame.

In some very compelling research, Anthony Andre looked at the design of automatic external defribulators (AEDs)(2003). An AED is a device used to resuscitate an individual experiencing cardiac arrest. AEDs are found in many public spaces, such as shopping malls, airports, and sporting events. An AED is intended to be used by the general public with no background or experience in life-saving techniques such as CPR. The design of an AED is critical, as most individuals who are actually using an AED are experiencing it for the first time, under a tremendous amount of stress. An AED must have simple and clear instructions, and deliver them in a way that is time sensitive and also mitigates user errors. Andre’s research compared four different AED manufacturers. He was interested in how each of them performed in terms of users being able to deliver a shock successfully within a specified time limit. He was also interested in identifying specific usability issues that were impacting user performance with each of the machines.

In his 2003 study, he assigned 64 participants to one of four different machines. Participants were asked to enter a room and save a victim (a mannequin lying on the floor) with the AED they were assigned. The results he found were shocking (no pun intended!). While two machines performed as expected (0% errors from a sample of 16 participants for each machine), two other machines did not fare so well. For example, 25% of the participants who used one of the AEDs were not able to deliver...