Assessing the Quality of Survey Data

Jörg Blasius, Victor Thiessen

• Publisher: SAGE Publications Ltd
• Year: 2012
• Print ISBN: 9781849203326, 9781849203319
• eBook ISBN: 9781446292327
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

This is a book for any researcher using any kind of survey data. It introduces the latest methods of assessing the quality and validity of such data by providing new ways of interpreting variation and measuring error. By practically and accessibly demonstrating these techniques, especially those derived from Multiple Correspondence Analysis, the authors develop screening procedures to search for variation in observed responses that do not correspond with actual differences between respondents. Using well-known international data sets, the authors exemplify how to detect all manner of non-substantive variation having sources such as a variety of response styles including acquiescence, respondents? failure to understand questions, inadequate field work standards, interview fatigue, and even the manufacture of (partly) faked interviews.

1

---

Conceptualizing data quality: Respondent attributes, study architecture and institutional practices

Assessing the quality of data is a major endeavour in empirical social research. From our perspective, data quality is characterized by an absence of artefactual variation in observed measures. Screening survey data means searching for variation in observed responses that do not correspond with actual differences between respondents. We agree with Holbrook, Cho and Johnson (2006: 569) who argue that screening techniques are essential because survey researchers are ‘far from being able to predict a priori when and for whom’ comprehension or response mapping difficulties will occur; and these are only two of many sources of poor data quality.

We think of data quality as an umbrella concept that covers three main sources affecting the trustworthiness of any survey data: the study architecture, the institutional practices of the data collection agencies, and the respondent behaviours. Study architecture concerns elements in the survey design, such as the mode of data collection (e.g., computer-assisted telephone interviews, mailed questionnaires, internet surveys), the number of questions and the order in which they are asked, the number and format of the response options, and the complexity of the language employed. Institutional practices cover sources of error that are due to the research organization, such as the adequacy of interviewer training, appropriateness of the sampling design, and data entry monitoring procedures. Data quality is obviously also affected by respondent attributes, such as their verbal skills or their ability to retrieve the information requested. While we discuss these three sources of data quality separately, in practice they interact with each other in myriad ways. Thus, self-presentation issues on the part of the respondent, for example, play a larger role in face-to-face interviews than in internet surveys.

While quality of data is a ubiquitous research concern, we focus on assessing survey data quality. Our concern is with all aspects of data quality that jeopardize the validity of comparative statistics. Group comparisons are compromised when the quality of data differs for the groups being compared or when the survey questions have different meanings for the groups being compared. If females are more meticulous than males in their survey responses, then gender differences that may emerge in subsequent analyses are suspect. If university-educated respondents can cope with double negative sentences better than those with less education, then educational differences on the distribution of such items are sub-stantively ambiguous. In short, it is the inequality of data quality that matters most, since the logic of survey analysis is inherently comparative. If the quality of the data differs between the groups being compared, then the comparison is compromised.

We further restrict our attention to the underlying structure of responses to a set of statements on a particular topic or domain. This topic can be a concrete object such as the self, contentious issues such as national pride or regional identity, or nebulous concepts such as democracy. Respondents are typically asked to indicate the extent of their acceptance or rejection of each of the statements. Their responses are expected to mirror their viewpoints (or cognitive maps as they will be called here) on that topic or issue. Excluded from consideration in this book is the quality of socio-demographic and other factual information such as a person’s age, income, education, or employment status.

In this chapter we first discuss the three sources of data quality, namely those attributable to the respondent, those arising from the study architecture, and those that emerge from inadequate quality control procedures of data collection agencies, including unethical practices. This is followed by a description of the nature and logic of our screening approach, which is anchored in scaling methods, especially multiple correspondence analysis and categorical principal component analysis. We conclude with a sketch of the key content of each of the subsequent chapters.

1.1 Conceptualizing response quality

---

We refer to sources of data quality that are due to respondents’ characteristics, such as their response styles and impression management skills, as response quality. Response quality is embedded in the dynamics common to all human interactions as well as the specific ones that arise out of the peculiar features of survey protocol. Common features, as recognized by the medical field, for example, emerge from the fact that a survey ‘is a social phenomenon that involves elaborate cognitive work by respondents’ and ‘is governed by social rules and norms’ (McHorney and Fleishman, 2006: S206).

The act of obtaining survey data imposes a particular stylized form of human interaction, which gives rise to its specific dynamics. The parameters that govern the survey form of interaction are as follows:

• The contact and subsequent interaction is initiated by the interviewer, typically without the express desire of the respondent.
• It occurs between strangers, with one of the members not even physically present when the survey is conducted via mail, telephone, or the web.
• The interaction is a singular event with no anticipation of continuity, except in longitudinal and/or other panel surveys where the interaction is limited to a finite series of discrete events.
• Interactional reciprocity is violated; specifically, the interviewers are expected to ask questions while the respondents are expected to provide answers.
• The researcher selects the complexity level of the language and its grammatical style, which typically is a formal one.
• The response vocabulary through which the respondents must provide their responses is extremely sparse.

In short, surveys typically consist of short pulses of verbal interaction conducted between strangers on a topic of unknown relevance or interest to the respondent, often in an alien vocabulary and with control of the structure of the interaction vested in the researcher. What the respondent gets out of this unequal exchange is assurances of making a contribution to our knowledge base and a promise of confidentiality and anonymity, which may or may not be believed. Is it any wonder, then, that one meta-analysis of survey data estimated that over half the variance in social science measures is due to a combination of random (32%) and systematic (26%) measurement error, with even more error for abstract concepts such as attitudes (Cote and Buckley, 1987: 316)? Clearly, these stylistic survey features are consequential for response quality. Such disheartening findings nevertheless form the underpinnings and the rationale for this book, since data quality cannot be taken for granted and therefore we need tools by which it can be assessed.

Given the features of a survey described above, it is wisest to assume that responses will be of suboptimal quality. Simon (1957) introduced the term ‘satisficing’ to situations where humans do not strive to optimize outcomes. Krosnick (1991, 1999) recognized that the survey setting typically induces satisficing. His application is based on Tourangeau and his associates’ (Tourangeau and Rasinski, 1988; Tourangeau, Rips and Rasinski, 2000) four-step cognitive process model for producing high-quality information: the respondent must (1) understand the question, (2) retrieve the relevant information, (3) synthesize the retrieved information into a summary judgement, and (4) choose a response option that most closely corresponds with the summary judgement. Satisficing can take place at any of these stages and simply means a less careful or thorough discharge of these tasks. Satisficing manifests itself in a variety of ways, such as choosing the first reasonable response offered, or employing only a subset of the response options provided. What all forms of satisficing have in common is that shortcuts are taken that permit the task to be discharged more quickly while still fulfilling the obligation to complete the task.

The task of responding to survey questions shares features with those of other literacy tasks that people face in their daily lives. The most important feature is that responding to survey items may be cognitively challenging for some respondents. In particular, responding to lengthy items and those containing a negation may prove to be too demanding for many respondents – issues that Edwards (1957) noted more than half a century ago. Our guiding assumption is that the task of answering survey questions will be discharged quite differently among those who find this task daunting compared to those who find it to be relatively easy.

Faced with a difficult task, people often use one of three response strategies: (1) decline the task, (2) simplify the task, and (3) discharge the task, however poorly. All three strategies compromise the response quality. The first strategy, declining the task, manifests itself directly in outright refusal to participate in the study (unit non-response) or failing to respond to particular items by giving non-substantive responses such as ‘don’t know’ or ‘no opinion’ (item non-response). Respondents who simplify the task frequently do this by favouring a subset of the available response options, such as the end-points of Likert-type response options, resulting in what is known as ‘extreme response style’. Finally, those who accept the demanding task may just muddle their way through the survey questions, perhaps by agreeing with survey items regardless of the content, a pattern that is known as an acquiescent response tendency. Such respondents are also more likely to be susceptible to trivial aspects of the survey architecture, such as the order in which response options are presented. We concur with Krosnick (1991) that response quality depends on the difficulty of the task, the respondent’s cognitive skill, and their motivation to participate in the survey. The elements of each of these are presented next.

Task difficulty, cognitive skills, and topic salience

---

The rigid structure of the interview protocol, in conjunction with the often alien vocabulary and restrictive response options, transforms the survey interaction into a task that can be cognitively challenging. Our guiding assumption is that the greater the task difficulty for a given respondent, the lower will be the quality of the responses given. Task characteristics that increase its difficulty are:

• numerous, polysyllabic, and/or infrequently used words;
• negative constructions (especially when containing the word ‘not’);
• retrospective questions;
• double-barrelled formulations (containing two referents but permitting only a single response);
• abstract referents.

Despite being well-known elements of task difficulty, it is surprising how often they are violated – even in well-known surveys such as the International Social Survey Program and the World Values Survey.

Attributes of the response options, such as their number and whether they are labelled, also contribute to task difficulty. Response options that are labelled can act to simplify the choices. Likewise, response burden increases with the number of response options. While minimizing the number of response options may simplify the task of answering a given question, it also diminishes the amount of the information obtained, compromising the quality of the data again. Formats that provide an odd number of response options are generally considered superior to even-numbered ones. This may be because an odd number of response options, such as a five- or 11-point scale, provides a mid-point that acts as a simplifying anchor for some respondents.

Whether the survey task is difficult is also a matter of task familiarity. The format of survey questions is similar to that of multiple choice questions on tests and to application forms for a variety of services. Respondents in non-manual occupations (and those with higher educational qualifications) are more exposed to such forms than their counterparts in manual occupations (and/or with less formal education). Public opinion surveys are also more common in economically developed countries, and so the response quality is likely to be higher in these countries than in developing countries.

Van de Vijver and Poortinga (1997: 33) point out that ‘almost without exception the effects of bias will systematically favor the cultural group from where the instrument originates’. From this we formulate the cultural distance bias hypothesis: the greater the cultural distance between the origin of a survey instrument and the groups being investigated, the more compromised the data quality and comparability is likely to be. One source of such bias is the increased mismatch between the respondent’s and researcher’s ‘grammar’ (Holbrook, Cho and Johnson, 2006: 569). Task difficulty provides another possible rationale for the cultural distance bias hypothesis, namely that the greater the cultural distance, the more difficult is the task of responding to surveys. The solution for such respondents is to simplify the task, perhaps in ways incongruent with the researcher’s assumptions.

Whether a task is difficult depends not only on the attributes of the task but also on the cognitive competencies and knowledge of the respondent, to which we turn next. Cognitive skills are closely tied to education (Ceci, 1991). For example, the research of Smith et al. (2003) suggests that elementary school children do not have the cognitive sophistication to handle either a zero-to-ten or a thermometer response format – formats that generally have solid measurement properties among adults (Alwin, 1997). Likewise, understanding that disagreement with a negative assertion is equivalent to agreement with a positively formulated one remains problematic even for some high school students (Marsh, 1986; Thiessen, 2010).

Finally, we assume that respondents pay greater heed to tasks on topics that interest them. Generally these are also the ones on which they possess more information and consequently also the issues for which providing a valid response is easier. Our approach to response quality shares certain features with that of Krosnick’s (1991, 1999) satisficing theory, which emphasizes the cognitive demands required to provide high-quality responses. For Krosnick, the probability of taking shortcuts in any of the four cognitive steps discussed previously decreases with cognitive ability and motivation, but increases with task difficulty. We agree that response optimizing is least prevalent among those with least interest or motivation to participate in a survey.

Normative demands and impression management

---

Surveys share additional features with other forms of verbal communication. First, the form of survey interaction is prototypically dyadic: an interviewer/researcher and a respondent in real or virtual interaction with each other. In all dyadic interactions, the members hold at least three images of each other that can profoundly affect the content of the viewpoints the respondent expresses: the image of oneself, the image of the other, and the image one would like the other to have of oneself. It is especially the latter image that can jeopardize data quality. Skilful interactions require one to be cognizant not only about oneself and the other, but also about how one appears to the other. Hence, the responses given are best conceived of as an amalgam of what respondents believe to be true, what they believe to be acceptable to the researcher or interviewer, and what respondents believe will make a good impression of themselves. Such impression management dynamics are conceptualized in the methodological literature as social desirability.

Second, we ordinarily present ourselves as being more consistent than we actually are. This is exemplified by comparing the determinants of actual voting in elections with those of reported voting. Typically the associations between various civic attitudes and self-reported voting behaviour are stronger than with actual (validated) voting behaviour (Silver, Anderson and Abramson, 1986). That is, our reported behaviours are more consistent with our beliefs than are our actual behaviours. If respondents initially report that they intended to vote, then they subsequently will be more likely to report that they voted even when they did not. Likewise, respondents who report that it is one’s civic duty to vote are more likely to report that they voted when they did not compared to those who did not think it was one’s civic duty.

Third, the normative structure places demands on the participants, the salience and extent of which depend on one’s social location in society. The social location of some respondents may place particular pressure on them to vote, or not to smoke, for example. These demand characteristics result in tendencies to provide responses that are incongruent with the positions actually held. The existing methodological literature also treats these pressures primarily under the rubric of social desirability, but we prefer the broader term of ‘impression management’. Van de Vijver and Poortinga (1997: 34) remind us that ‘[n]orms about appropriate conduct differ across cultural groups and the social desirability expressed in assessment will vary accordingly’.

It is precisely the existence of normative demands that required modifications to classical measurement theory. This theory provided a rather simple measurement model, whereby any individual’s observed score (y_i) is decomposed into two parts: true (τ_i) and error (ε_i); that is, y_i = τ_i + ε_i. While this formulation is enticingly simple, the problem emerges with the usual assumptions made when applied to a distribution. If one assumes that the error is uncorrelated with the true score, then the observed (total) variance can be...

Table of contents

• Cover Page
• Title Page
• Copyright
• Contents
• About the authors
• List of acronyms and sources of data
• Preface
• Chapter 1: Conceptualizing data quality: Respondent attributes, study architecture and institutional practices
• Chapter 2: Empirical findings on quality and comparability of survey data
• Chapter 3: Statistical techniques for data screening
• Chapter 4: Institutional quality control practices
• Chapter 5: Substantive or methodology-induced factors? A comparison of PCA, CatPCA and MCA solutions
• Chapter 6: Item difficulty and response quality
• Chapter 7: Questionnaire architecture
• Chapter 8: Cognitive competencies and response quality
• Chapter 9: Conclusion
• References
• Index