Credible Checklists and Quality Questionnaires

A User-Centered Design Method

Chauncey Wilson

• Publisher: Morgan Kaufmann
• Year: 2013
• ISBN:
  9780124104495
• Topic: Ciencia de la computación
• Subtopic: Interacción persona-computadora

Credible Checklists and Quality Questionnaires starts off with an examination of the critical but commonly overlooked checklist method.

In the second chapter, questionnaires and surveys are discussed. Asking questions sounds simple, but the hard truth is that asking questions (and designing questionnaires) is a difficult task. This chapter discusses being mindful of the choice of words, order of questions and how early questions influence later questions, answer scales and how they impact the user response, questionnaire design, and much more.

The final chapter provides examples of some common questionnaires (both free and fee-based) for assessing the usability of products.

After reading this book, readers will be able to use these user design tools with greater confidence and certainty.

Questionnaires and Surveys

This book begins with a brief description of the essential, but often overlooked checklist method. Checklists are used for many things in user-centered design (UCD), such as verifying that we are prepared for a usability test, that we have asked the critical questions during field interviews, and that we have followed best practices in the design of questionnaires. Chapter 1 provides some basic principles and practical tips for creating good checklists for UCD activities. Chapter 2 can be viewed as a checklist for questionnaire and survey designers. At the end of the book, an appendix provides information on some well-known usability questionnaires that might be useful for readers of this book.

Keywords

Checklist; checklist guidelines; questionnaire; survey; questions; usability; questionnaires guidelines; standard questionnaires; questionnaire bias; questionnaire best practices; user experience

The important thing is not to stop questioning. Curiosity has its own reason for existing.

–Albert Einstein

Overview of Questionnaires and Surveys

When Should You Use Questionnaires and Surveys?

Strengths

Weaknesses

What Do You Need to Use Questionnaires and Surveys?

Personnel, Participants, and Training

Software

Documents and Materials

Procedures and Practical Advice

Planning and Developing the Questionnaire and Survey

Conducting the Questionnaire Study

Post-Survey Activities

Varying and Extending Questionnaires and Surveys via Mixed-Mode Surveys

Major Issues in the Use of Questionnaires and Surveys

Using Questionnaires to Compare Products Against a Norm

Wording of Questions

Questionnaire and Question Length

Recall and Memory

Ordering of Questions

Guidelines for Designing Response Categories

Rating Scales

Arbitrary Scaling

Likert Scaling

Semantic Differential Scales

Thurstone Scaling (Equal-Appearing Intervals)

Questionnaire Layout

Rewards and Incentives

Data Analysis

Types of Data

Analysis Techniques

Qualitative Analysis of Open-Ended Data

Exploratory Data Analysis

Frequency Counts

Measures of Central Tendency, Variability, and Shape

Cross Tabulations

References

Alternate Names: Polls, questionnaire study

Related Methods: Critical incident technique, diary method, phone interview, semi-structured interview, structured interview

Overview of Questionnaires and Surveys

A questionnaire is a written, online, or verbal tool for collecting data from individuals or groups that can be analyzed using qualitative and quantitative techniques. Building on the concepts introduced in Chapter 1, this chapter serves as a checklist on how to develop questions, questionnaires, and surveys. Developing a questionnaire is only one aspect of a survey. Other aspects of survey design include the following:

• Devising a sampling plan: Who are the respondents and how many respondents do you need to provide reliable and valid results? If you have a bad sample, your data might lead the product team astray.

• Developing a data analysis plan: What types of data will help you meet your goals? What analyses will help you understand the raw data?

• Recruiting respondents: Recruiting the target respondents is often a struggle, especially with privacy, opt-out policies, anti-spam policies, and difficulties getting directly at users (as opposed to customers).

• Writing cover letters and introductions that explain your study: Good cover letters can influence response rates.

• Sending out follow-up reminders: Reminders can be quite effective at encouraging responses, but they need to be written to be both gentle and persuasive so the respondent is not irritated.

• Analyzing the data: For large-scale studies, it is useful to consult with a business analyst or statistician who can guide you through data analysis issues such as inflated alpha levels (when you do multiple statistical tests on the same set of data, you are more likely to find spurious significant results) and confidence intervals (a way to assess the uncertainty due to sampling around statistics, such as means).

• Interpreting the data: What do your data mean? Interpretation involves both art and science. You need to understand how the sampling and context of your study can affect your results.

• Presenting the data to stakeholders: What is the best approach for presenting your data? How much do you present? How much detail do you need to ensure credibility?

This chapter focuses on the fundamentals of questionnaire design, practical tips, and some selected topics related to the broader survey process. A large collection of research on how to design, conduct, and analyze surveys (Bradburn, Sudman, & Wansink, 2004; Converse & Presser, 1986; Dillman, Smyth, & Christian, 2009; Foddy, 1993; Groves et al., 2009) is available, but the research can be confusing, contradictory, and complex. The intent of this chapter is to provide some reasonable practical advice, based on many years of reviewing and improving questionnaires, and not to debate in depth whether a five-point scale is necessarily better or worse than a seven-point scale.

The Long History of Surveys and Questionnaires

One of the earliest surveys on record is described in the Domesday Book, which reported on a land survey commissioned in 1085 by William the Conqueror to determine who owned land and other resources in England. The survey results were used to determine how much tax could be extracted from the landowners.

Around 800 years later, in the 1870s, Sir Francis Galton, the Victorian polymath and cousin of Charles Darwin, introduced the modern questionnaire method into psychology with his controversial book, English Men of Science: Their Nature and Nurture (Galton, 1874). Galton used the questionnaire method to examine whether interest in science was due to heredity or environment by sending questionnaires to 190 members of the Royal Society and asking questions about environment, birth order, and characteristics of the parents of Society members.

A questionnaire is often viewed as an easy way to gather data quickly from many respondents. However, the common belief that questionnaires are easy to design (often expressed as “let us just throw a questionnaire together” by those with limited questionnaire design experience) is a myth. The design of questionnaires and surveys is a complex process that involves many, sometimes conflicting, considerations (Dillman, 2000, 2007; Dillman et al., 2009; Kirakowski, 2000; Nemeth, 2004; Robson, 2002; Sudman, Bradburn, & Schwarz 1996), including some basic assumptions:

• The questionnaire is tapping the concepts of interest.

• The potential respondents are competent sources of the information you need.

• The respondents are willing to commit the time to answer your questionnaire (this gets into relevance and extrinsic and intrinsic motivation).

• All of the respondents understand your questions in the same way.

• People will follow the rules of the survey and provide (to the best of their abilities) honest data.

This chapter describes the basic procedures for developing paper and online questionnaires and highlights some of the important considerations for designing questionnaires and conducting surveys, including the following:

• The structure of questions: When should you use open, closed, or mixed questions? Mixed questions include both specific responses and an open-ended “Other” response...