Reliability and Validity in Qualitative Research

Jerome Kirk, Marc L. Miller

• Publisher: SAGE Publications, Inc
• Year: 1985
• Print ISBN: 9780803924703
• eBook ISBN: 9781506319322
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

Authors Jerome Kirk and Marc L. Miller define what is—and what is not—qualitative research. They suggest that the use of numbers in the process of recording and analyzing observations is less important than that the research should involve sustained interaction with the people being studied, in their own language and on their own turf. Following a chapter on objectivity, the authors discuss the role of reliability and validity and the problems that arise when these issues are neglected. They present a paradigm for the qualitative research process that makes it possible to pursue validity without neglecting reliability.

RELIABILITY AND VALIDITY IN QUALITATIVE RESEARCH

JEROME KIRK
University of California, Irvine

MARC L. MILLER
University of Washington

AUTHORS’ NOTE: We relish this opportunity to apologize to all those we fear to acknowledge.

1. OBJECTIVITY IN QUALITATIVE RESEARCH

Qualitative research is a particular tradition in social science that fundamentally depends on watching people in their own territory and interacting with them in their own language, on their own terms. As identified with sociology, cultural anthropology, and political science, among other disciplines, qualitative research has been seen to be “naturalistic,” “ethnographic,” and “participatory.”

It should be remarked at the outset that the term “qualitative” in reference to this tradition has led to a variety of misunderstandings. Technically, a “qualitative observation” identifies the presence or absence of something, in contrast to “quantitative observation,” which involves measuring the degree to which some feature is present. To identify something, the observer must know what qualifies as that thing, or that kind of thing. This entails counting to one. It follows from this narrow consideration that qualitative research would denote any research based on percentages, means, chi-squares, and other statistics appropriate to cardinal, or counting, numbers.

On the other hand, “quality” connotes the nature, as opposed to the “quantity,” or amount, of a thing. According to this equally limited consideration, qualitative research would denote any research distinguished by the absence of counting.

These two plausible definitions directly contradict one another. Neither suits the present purpose very well. Whether or not a number gets used in the process of recording and analyzing observations is an entirely abstract issue. By our pragmatic view, qualitative research does imply a commitment to field activities. It does not imply a commitment to innumeracy. Qualitative research is an empirical, socially located phenomenon, defined by its own history, not simply a residual grab-bag comprising all things that are “not quantitative.” Its diverse expressions include analytic induction, content analysis, semiotics, hermeneutics, elite interviewing, the study of life histories, and certain archival, computer, and statistical manipulations. One purpose of the series of which this volume is a part is to elaborate on these and other possibilities.

The accumulated wisdom of the academic tradition of qualitative research is largely a formal distillation of sophisticated techniques employed by all sorts of professionals—adventurers, detectives, journalists, spies—to find out things about people. Necessarily, the formal tradition has been accompanied by certain distinctive orientations. Qualitative research is socially concerned, cosmopolitan, and, above all, objective.

Objectivity

“Objectivity,” too, is an ambiguous concept. In one sense, it refers to the heuristic assumption, common in the natural sciences, that everything in the universe can, in principle, be explained in terms of causality. In the social sciences, this assumption often seems to miss the point, for much of what social scientists try to explain is the consequence of inner existential choices made by people. In ordinary language, when we ask “why” a person acts as he or she does, we are generally inquiring teleologically about his or her purposes. Indeed, if knowledge itself is taken to be merely the inevitable consequence of some mechanistic chain of cause and effect, its logical status would seem to be compromised.

In another sense, “objectivity” refers to taking an intellectual risk—the risk of being demonstrably wrong. Popper’s model of the hypothetico-deductive method exemplifies this connotation. According to Popper (1959: 42), the scientist prepares to test theories by deriving from them hypotheses that can in principle break down when applied in the real world:

 

What characterizes the empirical method is its manner of exposing to falsification, in every conceivable way, the system to be tested. Its aim is not to save the lives of untenable systems but, on the contrary, to select the one which is by comparison the fittest, by exposing them all to the fiercest struggle for survival.

Even in Popper’s sophisticated formulation, the hypothetico-deductive model is rather an inaccurate and schoolmarmish description of what scientists do, but it properly contrasts the scientific enterprise with others (such as art or ethics) in which practitioners do not routinely subject their theories to that sort of empirical risk, or their egos to the potential of battery not only by the arguments of intellectual adversaries but also by the demonstrative refutation of the empirical world.

It is in this latter sense that qualitative researchers have always celebrated objectivity. A commitment to objectivity does not imply a desire to “objectify” the subject matter by “over-measurement” (Etzioni, 1964), or to facilitate authoritarian social relationships by treating human beings as though they were certain features they may happen to have.¹ It does not presuppose any radically positivist view of the world; it emphatically eschews the search for final, absolute “truth,” preferring to leave such an enterprise to philosophers and theologians.

The assumptions underlying the search for objectivity are simple. There is a world of empirical reality out there. The way we perceive and understand that world is largely up to us, but the world does not tolerate all understandings of it equally (so that the individual who believes he or she can halt a speeding train with his or her bare hands may be punished by the world for acting on that understanding). There is a long-standing intellectual community for which it seems worthwhile to try to figure out collectively how best to talk about the empirical world, by means of incremental, partial improvements in understanding. Often, these improvements come about by identifying ambiguity in prior, apparently clear, views, or by showing that there are cases in which some alternative view works better. Previously held views are not in general taken to be refuted by such contributions, but complemented by them. “Truth” (or what provisionally passes for truth at a particular time) is thus bounded both by the tolerance of empirical reality and by the consensus of the scholarly community (Blumer, 1968).

Natural science is strongly identified with a commitment to objectivity. Like natural science, qualitative social research is pluralistic. A variety of models may be applied to the same object for different purposes. A man may be an object of a certain mass and size to an engineer, a bundle of neuroses to the psychologist, a walking pharmacy to the biochemist, and a bank account with desires to an economist. Light may have a frequency or (in this case, by a describable transformation) consist of photons. Water is the canonical acid and the ultimate primitive base. Natural human vision is binocular, for seeing the same thing simultaneously from more than one perspective gives a fuller understanding of its depth. The reason Einstein originally called his theory of relativity the Theory of Invariance is because though everything displays different aspects to different viewpoints, some features remain the same.

Plan of This Book

The several points in this orientation are easily reviewed. Qualitative research is a sociological and anthropological tradition of inquiry. Most critically, qualitative research involves sustained interaction with the people being studied in their own language, and on their own turf. Less important is whether or not, or at what level of sophistication, numbers are employed to reveal patterns of social life. To see qualitative research as strictly disengaged from any form of counting is to miss the point that its basic strategy depends on the reconciliation of diverse research tactics.

It is our view that qualitative research can be performed as social science. Understanding the workings of a scientific endeavor, whether it is of the natural or social variety, entails an appreciation of its objectivity. By this convention, the objectivity of a piece of qualitative research is evaluated in terms of the reliability and validity of its observations—the two concepts to which this monograph is devoted.

Chapter 2 introduces the role of reliability and validity in the unfolding of science. Chapter 3 more fully explores the meaning of validity, and points out that much research (particularly nonqualitative research) lacks not only validity but also any means of appraising its validity. In Chapter 4, the history of qualitative research is seen as a cumulative effort to correct this flaw: Were it otherwise valueless, qualitative research would be justified solely as a validity check. Yet, as is pointed out in Chapter 5, much of the validity of qualitative research has been gained at the expense of reliability in the “discovery,” or data-collection phase of research. Finally, Chapter 6 presents a model of the fieldwork activity that constitutes discovery in qualitative research, and provides some detailed instructions for maintaining reliability in the process.

2. RELIABILITY AND VALIDITY

Despite the prestige and success of natural science in recent years, application of science as a model for social “science” is not inevitable. Many have argued that social science has an intrinsically different set of goals that call for an altogether separate collection of methods. Others (nonscientists, it should be noted) contend that recent developments in the natural sciences entirely discredit the fundamental notions (such as objectivity) of an earlier and outdated science.

Yet, whatever their detailed goals, the natural and social sciences share an aspiration to cumulative collective knowledge that is of interest on its own merits to those other than the friends and admirers of its creators. This goal is exactly objectivity. In the natural sciences, objectivity is obtained in two ways. First, experience is reported in such a way that it is accessible to others, for example, when reporting an experiment every effort is made to describe the way the experiment was carried out, just in case somebody else would like to try the same thing. Second, the results of the experiment are reported in terms of theoretically meaningful variables, measured in ways that are themselves justifiable in terms of the relevant theories.

Since Wilhelm Dilthey and George Herbert Mead, the vast majority of social scientists have agreed that objectivity, in this sense, is an admirable goal. Yet, the description of reliability and validity ordinarily provided by nonqualitative social scientists rarely seems appropriate or relevant to the way in which qualitative researchers conduct their work.

It is the purpose of this book to reconcile the means-ends discrepancy. The remainder of this chapter will pursue the argument that, subject to clearly specifiable differences in goals and practice, social science is in every sense of the word fully as “scientific” as physics, and has fully as much need for reliability and validity as any other science.

The “Positivist” View

In recent decades, the social science literature has incorporated a great deal of discussion of an epistemology called “positivism.” (The term is generally employed by those advocating some alternative view of knowledge, and often amounts to a straw man.) In its strongest form, positivism denies objectivity as defined here by assuming not only that there is an external world, but that the external world itself determines absolutely the one and only correct view that can be taken of it, independent of the process or circumstances of viewing. No one seriously defends such an ontology, but scholars attentive to the social and cultural construction of social things (including social science) point out that much research (particularly nonqualitative research) makes sense only in terms of a set of unexamined positivist assumptions.

Most often, these assumptions pertain to the “naturalness” of the measurement procedure employed. Thus a survey researcher may interview a large number of people about their political attitudes, and conclude that “public opinion” says something. Such an assertion obviously concerns the investigator’s theoretical view of the world as much as it does the psychic organization of the interviewees. The investigator’s theory contains categories not imposed by the structure of empirical reality. Elements such as “attitudes” and “public opinion” serve rather to organize understanding of the world. Certainly, political and psychological theories that do not use these constructs (or even deny their meaningfulness) are possible, and treating analytic devices as though they are facts is the well-known fallacy of reification.

In response to the propensity of so many nonqualitative research traditions to use such hidden positivist assumptions, some social scientists have tended to overreact by stressing the possibility of alternative interpretations of everything to the exclusion of any effort to choose among them. This extreme relativism ignores the other side of objectivity—that there is an external world at all. It ignores the important distinction between knowledge and opinion, and results in everyone having a separate insight that cannot be reconciled with anyone else’s.

Metaphysical polemics, often directed against caricatures of the opposing views, largely miss the point. As is shown in the next chapter, the problem is not so much one of metaphysics as it is a pragmatic question of the validity of measurements. The survey researcher who discusses attitudes is not wrong to do so. Rather, the researcher is wrong if he or she fails to acknowledge the theoretical basis on which it is meaningful to make measurements of such entities and to do so with survey questions addressed to a probability sample of voters.

For any observation (or measurement) to yield discovery, it must generate data that is (a) not already known and (b) identifiable as “new” by the theory already in place.² Most of the technology of “confirmatory “nonqualitative research in both the social and natural sciences is aimed at preventing discovery. When confirmatory research goes smoothly, everything comes out precisely as expected. Received theory is supported by one more example of its usefulness, and requires no change. As in everyday social life, confirmation is exactly the absence of insight. In science, as in life, dramatic new discoveries must almost by definition be accidental (“serendipitous”). Indeed, they occur only in consequence of some kind of mistake.

The Discovery of the New

Henri Becquerel was studying the phenomenon of phosphorescence by exposing metal salts first to the sun and then to photographic plates. When the sky clouded over for an extended period, he tossed the uranium salts into a drawer with his photographic materials and knocked off work for a while (Badash, 1965). “Merde! Je me suis planté!” he must have muttered when he discovered that the film was ruined, but he was sufficiently prepared and alert to realize that he had discovered radioactivity.

More recently, the first men to hear the echo of the origin of the universe thought they were listening to guano. In 1964, Arno Penzias and Robert Wilson set out to measure the radio waves emitted from different latitudes in our disk-shaped galaxy. First, they had to identify what portions of the signal they received originated in the instrument itself. When they received a strong signal in microwave frequencies (where galaxies emit virtually no radiation), their first move was to devote considerable time and expense to cleaning out the “white dielectric material” deposited by pigeons in the antenna throat. This produced o...

Table of contents

• Cover page
• Title
• Copyright
• Contents
• Series Introduction
• Editors’ Introduction
• 1. Objectivity in Qualitative Research
• 2. Reliability and Validity
• 3. The Problem of Validity
• 4. Toward Theoretical Validity
• 5. The Problem of Reliability
• 6. Ethnographic Decision Making: The Four Phases of Qualitative Research
• Notes
• Glossary
• References
• About the Authors