Data Collection

Planning for and Collecting All Types of Data

Patricia Pulliam Phillips, Cathy A. Stawarski

• Publisher: Pfeiffer
• Year: 2016
• Print ISBN: 9780787987183
• eBook ISBN: 9781119254782
• Edition: 1
• Topic: Business
• Subtopic: Human Resource Management
• Index: Business

Data Collection Data Collection is the second of six books in the Measurement and Evaluation Series from Pfeiffer. The proven ROI Methodology--developed by the ROI Institute--provides a practical system for evaluation planning, data collection, data analysis, and reporting. All six books in the series offer the latest tools, most current research, and practical advice for measuring ROI in a variety of settings. Data Collection offers an effective process for collecting data that is essential to the implementation of the ROI Methodology. The authors outline the techniques, processes, and critical issues involved in successful data collection. The book examines the various methods of data collection, including questionnaires, interviews, focus groups, observation, action plans, performance contracts, and monitoring records. Written for evaluators, facilitators, analysts, designers, coordinators, and managers, Data Collection is a valuable guide for collecting data that are adequate in quantity and quality to produce a complete and credible analysis.

1
Using Questionnaires and Surveys

Data collection is the first operational part of the ROI process model. Data are collected in different time frames and from different sources. This is the first of four chapters on data collection methods. Collectively, these four chapters will provide a variety of ways to meet any application, budget, or time constraint.

Probably the most common data collection method is the questionnaire (Alreck and Settle, 1995). Ranging from short reaction forms to detailed follow-up tools, questionnaires can be used both to obtain subjective information about participants and to document objective, measurable impact results for an ROI analysis. Because of this versatility, the questionnaire is the preferred method for capturing data at Levels 1, 2, 3, and 4 in some organizations.

A survey is a specific type of questionnaire with several applications in measuring program success. Surveys are used in situations in which only attitudes, beliefs, and opinions are captured; questionnaires are much more flexible, capturing a wide range of data from attitudes to specific improvement statistics. The principles of survey construction and design are similar to those of questionnaire design. This chapter explains how to develop both types of instruments.

Types of Questions

In addition to the types of data sought, the types of questions distinguish surveys from questionnaires. Surveys may elicit yes-or-no responses, if absolute agreement or disagreement is required, or they may solicit a range of responses, often on a five-point scale from “strongly agree” to “strongly disagree.”

A questionnaire may contain any or all of these types of questions:

• Open-ended questions allow unlimited answers. Questions are followed by ample blank space for the responses.
• Checklists provide a list of items, and the participant is asked to check those that apply in the situation.
• Two-way questions limit answers to a pair of alternative responses, such as yes and no.
• Multiple-choice questions provide several possible answers, and the participant is asked to select the one that is most applicable.
• Ranking scales require the participant to rank a list of items.

Questionnaire Design Steps

Nothing is more confusing, frustrating, and potentially embarrassing than a poorly designed or improperly worded questionnaire. Fortunately, with thought and planning, these problems can be easily avoided. Questionnaire design is a logical process that can be divided into simple steps. Use the following steps to help you develop a valid, reliable, and effective instrument (Robson, 2002).

Determine the Specific Information Needed

The first step in questionnaire design is reviewing the objectives, topics, skills, or attitudes presented in the program for potential questionnaire items. Developing this information in outline form is sometimes helpful so that related questions or items can be grouped. At this time, also explore issues related to the application and impact of the program for inclusion in the questionnaire.

Involve Stakeholders in the Process

To the extent possible, stakeholders—clients, sponsors, supporters, or other interested parties—should be involved in the questionnaire design process. Ask those most familiar with the program to provide information on specific issues and concerns that might affect how the actual questions are framed for the questionnaire. In some cases, stakeholders may want to provide input on specific issues or items. Not only is stakeholder input useful in questionnaire design but it also builds ownership in the measurement and evaluation process and supports content validity.

Select the Types of Questions

From the five types of questions described previously, select the type or types that will result in the specific data needed. The planned data analysis and variety of data needed should be considered when deciding which types of questions to use.

Develop the Questions

The next step is to develop specific questions based on the type of questions selected and the information needed. Questions should be simple and straightforward, to avoid confusing the participants or leading them toward a desired response. Each question should address only one issue. If multiple issues need to be addressed, divide questions into multiple parts or develop separate questions for each issue. Avoid terms or expressions that might be unfamiliar to participants.

Check the Reading Level

To ensure that the questionnaire can be easily understood by the target audience, assess the reading level of the questionnaire. Many word processing programs have a function that can determine the reading difficulty of a text, indicating what grade level of education would be needed to read it. This important check ensures that the reading level of the questionnaire matches that of the target audience.

Test the Questions

Proposed questions should be tested to make sure that they will be correctly understood. Ideally, the questions should be tested on a sample group of participants. If this is not feasible, the sample group of employees should be at approximately the same job level as the participants. Seek feedback, critiques, and suggestions from the sample group so that the questionnaire design can be improved before it is administered to participants. Ensure that questions reflect program objectives and content.

Address the Anonymity Issue

Participants must feel free to respond openly to questions, without fear of reprisal. The confidentiality of their responses is of the utmost importance because there is usually a link between a questionnaire’s anonymity and respondents’ honesty. Therefore, questionnaires and surveys should be anonymous unless individuals must be identified for specific reasons. In situations in which participants must complete the questionnaire as a captive audience or submit a completed questionnaire directly to an individual, have a neutral third party collect and process the data, to ensure that participants’ identities are not revealed. In cases in which individual identities must be known (for example, to compare output data with previous data or to verify the data), make every effort to keep respondents’ identities from being revealed to those who might be biased by their responses. Confidentiality goes a long way when collecting data for current or future evaluation projects.

Design for Ease of Tabulation and Analysis

Consider how each potential question will affect data tabulation, data summary, and data analysis. If possible, outline and review the data analysis process at this point. This step will help you avoid problems of inadequate, cumbersome, or lengthy data analysis caused by improper wording or design of questionnaire items.

Develop the Completed Questionnaire and Prepare a Data Summary

Integrate and develop the questions into an attractive questionnaire with instructions that will allow it to be administered effectively. In addition, develop an analysis spreadsheet so that the data can be tabulated quickly for analysis. Developing the questionnaire and planning data analysis in tandem will ensure appropriate and efficient reporting in the end.

Improving the Response Rate for Questionnaires and Surveys

The items on a questionnaire represent a wide range of potential issues to explore. Obviously, asking all of the possible questions could result in a reduced response rate. The challenge, therefore, is to design and administer the questionnaire so as to maximize the response rate while ensuring reliable responses. Asking too many questions can reduce the number of questionnaires returned; asking too few can negatively affect the reliability of the results. Response rate management is a critical issue if the questionnaire is the primary data collection method and most of the evaluation hinges on the response of the participants. Taking the actions discussed in this section can help increase response rates.

Provide Advance Communication

If it is appropriate and feasible, communicate with participants in advance about the requirement of completing the questionnaire. Advance warning reduces some of the resistance to the process, provides an opportunity to explain in more ...

Table of contents

• Cover
• Table of Contents
• About This Book
• The Measurement and Evaluation Series
• Introduction to the Measurement and Evaluation Series
• About Pfeiffer
• Title
• Copyright
• Acknowledgments from the Editors
• Principles of the ROI Methodology
• 1 Using Questionnaires and Surveys
• 2 Using Tests
• 3 Using Interviews, Focus Groups, and Observation
• 4 Using Other Data Collection Methods
• 5 Measuring Reaction and Planned Action
• 6 Measuring Learning and Confidence
• 7 Measuring Application and Implementation
• 8 Measuring Impact and Consequences
• 9 Selecting the Proper Data Collection Method
• Index
• About the Authors
• Pfeiffer Publications Guide
• What will you find on pfeiffer.com?
• Measurement and Evaluation Series
• Special Offer from the ROI Institute
• End User License Agreement