Reviewing Qualitative Research in the Social Sciences

Audrey A. Trainor, Elizabeth Graue, Audrey A. Trainor, Elizabeth Graue

• Publisher: Routledge
• Year: 2013
• ISBN:
  9781136699238
• Edition: 1
• Topic: Education
• Subtopic: Research in Education

Foundational characteristics of qualitative research include flexibility, variation in application, critique, and innovation all of which derive from its subjective roots in interpretivism and constructivism. While the scholars who design qualitative research projects envision these qualities as strengths, such a breadth of practices and the assumptions that undergird them may present challenges during the peer review process. As a result, those who review and consume qualitative research often have important and difficult-to-answer questions about the project's design, strategies/tools, and analysis, with few guidelines for gauging the merit of the work.

The mission of this book is to provide a useful guide for researchers, reviewers, and consumers who are charged with judging the quality of qualitative studies. In order to embrace the challenges and controversies that accompany this goal, the editors have solicited experts representing multiple disciplines and methods of qualitative inquiry. Their contributions represent the rich diversity in the field while simultaneously producing a pragmatic and useful guide. While it is neither possible nor desirable to compartmentalize qualitative approaches and issues into neatly organized categories, the construct of method has been chosen as a common organizing device.

The introductory chapter explains the need for such a book and underscores the foundational strengths of qualitative research: flexibility, variation, critique, and innovation. The remaining chapters review the principal approaches to qualitative research with care taken not to standardize, rigidly define, or oversimplify any approach. For ease of use, all methodological chapters are organized around the following elements of inquiry which reviewers tend to examine: definition, sampling, data collection, data analysis, representation, and congruency.

1

Introduction

Audrey A. Trainor and Elizabeth Graue

The room is packed—surprising for a session scheduled first thing in the morning. Audience members are jockeying for seats and the panel is chatting as they figure out the order of presentations. At 8:15 a.m. on the dot the chair of the session welcomes the group.¹

Dr. Intro: I'd like to welcome you all to what I think will be a very exciting session today. As the title indicates, the topic is Quality in Qualitative Research. The crowd that has gathered today is an indication of the broad interest in this topic and I'm looking forward to a stimulating conversation. It was arranged by the head of the qualitative interest group as a way to publicly talk through an idea presented to the board last year—that we develop a reviewer's guide for qualitative research. We've gathered together a group of thinkers representing varied perspectives on qualitative research and posed the question of whether developing this kind of guide will enrich our work or constrain it. Why don't we begin with Dr. Lovett?

Dr. Lovett: I love the idea of this text. I teach a course on reading social science research, and having a text that could walk students through the processes of judging the quality of the research would be invaluable. Also I edit a journal, and some of our reviewers are not from social science fields, but they are reading and reviewing social science papers, and I think some of them would be very pleased to have a good resource to help guide them through the process. I also frequently chair dissertation studies and would love to have a resource that could help students understand and evaluate the work that they are reading; I think it would help make them more critical and selective. In short, I'm pretty much whom you'd think of to buy, and recommend, this book.

Prof. Subjective: I think successfully accomplishing this task may be more challenging than you think. By its very nature, qualitative approaches to research are varied, flexible, critical, and innovative. The idea that one might reduce each sub-discipline to a series of checklists would likely offend the very individuals for whom the book is meant to provide support. In contrast, should the book encourage “non-qualitative” reviewers to “let go” and embrace the variety within approaches, the potential consumers of the book will likely be disappointed.

Dr. Lovett: The key will be to think about what reviewers would need to know about qualitative research to make informed decisions about a study's quality. I worry that authors most likely will write methods chapters for researchers rather than informational chapters for reviewers.

Prof. Careful: I agree—the audience is paramount. My confusion is whether the book will be read as a “how to” manual for novice researchers or a guide for reviewers. My sense is that it would be difficult to create an outstanding guide for reviewers that would also serve as an outstanding teaching guide—while the content overlaps the orientation to the text and reader would probably be different so as to serve its different purposes. A text that would guide reviewers would provide examples of how terminology might be used differently in different designs for the purpose of helping them make a decision as to appropriateness, rather than to elicit a critical discussion, which a teaching oriented text might do. Because a good book to guide reviewers is lacking and there are plenty of introductory texts on qualitative research, I would encourage the editors to keep the review orientation in the forefront when providing guidelines to chapter authors.

Prof. Intro: Thanks—so far it seems that we're hearing that there is a need for this resource, with the caveat that doing it well will be difficult. I'd love to hear more from you Dr. Subjective.

Dr. Subjective: I may be the lone voice of dissent but I am not sure that this is a project that is doable. Qualitative research ultimately reflects a subjectivist/subjective relationship to knowledge. Those unwilling to embrace qualitative research do so given a lack of trust and respect for the subjective nature of the approach, a decision that ultimately precedes one's respectful consideration of rigor and method. In short, qualitative research is less published because some reviewers do not trust it. I doubt a volume that argues for the flexibility of design would provide reviewers with any more trust in the approach. Similarly, a volume that reduces qualitative methodologies to a prescriptive set of structures would not be well aligned with the principles of the approach.

Dr. Who are We?: I understand Dr. Subjective's position—does that mean we have intersubjective agreement? (laughter) I think the success of a project like this will depend on the degree to which you keep the focus on evaluating quality within methods, under different theoretical and philosophical orientations. Here's my concern—when do you call something methods, methodologies, or theoretical frameworks? I know that this sounds like a simple intro to qual methods question, but it has important implications for the organization and conceptualization of this kind of book. So if I brainstorm a list of chapters, I end up with a muddled mess. Some are methodologies (action research, case study, mixed methods), some are theoretical perspectives (critical theory, phenomenology), and some are methods (GT, DA, etc). Unlike a handbook for quantitative methods, qualitative researchers do not share a single epistemology—ontological and epistemological views have major implications for assessing “quality.” Any approach (focusing on analysis methods, methodologies, or even theoretical perspectives) becomes problematic because the number of possible epistemological/ methodological combinations (constructivist study with interviews and narrative analysis, constructivist study with visuals and semiotic analysis, etc.) is almost endless. I am not sure how to address, discuss, or illustrate this type of variance and diversity. I also wonder what enough detail is for this type of audience—how much nuances are needed and/or helpful.

Prof. Intro: This is a classic problem—and one that is in dispute if you compare discussions across methodology texts. What are your thoughts on what should be included in such a text? What organizational structure will support the key ideas?

Dr. Who are We?: There are so many ways to cut the chapters—by methods, by various design elements rather than methods, around the more commonly used orientations and focus on how quality would be defined for different types of activities of different types of studies.

Prof. Careful: How you define the structure is all bound up with what you want it to be and with whom you would like to talk. The main issue is developing a structure that speaks to qualitative researchers who are writing the chapters but also helps readers unfamiliar with the terrain navigate the vast amount of terms that each tradition brings with it.

Dr. Subjective: If the goal is to support qualitative researchers in gaining access to research support from a new set of organizations, it seems to me that a more important contribution would be in the form of a book that helps qualitative researchers understand the language and values of these organizations. Rather than asking these institutions to change for us, perhaps it would be easier if we figured out how to communicate our findings to them in a way that they would view as more applicable. Translational research is something that is very “hot” in the large, scientific funding circles. Qualitative research in many forms seems very well suited to the demands and language of this paradigm. Wouldn't it make more sense to think about how qualitative researchers may frame their work in the language of a translational researchers, rather than trying to get NIH to change the way it thinks about the value of “performance” research?

Prof. Intro: So it seems that we are back to the question of meaning—how qualitative—what are the goals for this project, what ends are we working toward, and what are the appropriate means we'd use to get there. We've heard a variety of ideas this morning and it looks like the audience is ready to weigh in on the topic. Let's open things up for questions and comments . . .

Embarking on Unchartered Waters

When we first came together to discuss the possibility of a reviewer's guide, we thought such an endeavor would be a challenge, but we perhaps misjudged the extent to which other qualitative researchers would react, either with a healthy dose of skepticism or with enthusiasm, to the idea. But, really, how should we have known? Methods books sometimes include criteria for quality, and over the years, experts have provided the field with discussions about criteria (see, for example, Brantlinger et al., 2005; Caelli et al., 2003; Lincoln & Guba, 1986; Lincoln, 1995; Morrow, 2005). Yet no similar volume directed toward the art of reviewing exists as a thumbnail for comparison in the field of qualitative methods. In retrospect, of course, the absence of such should have been our first clue that the project would spark lively feedback from enthusiasts and skeptics alike.

The introductory script, constructed from our reflections and excerpts of the six reviews of the proposal for this volume, illustrates the divergent nature of the range of expert reviewers’ opinions, matched by the intensity of the sentiments behind them. Obviously, albeit with tentative footing, we were able to advance our belief that such a volume could contribute to the field, and we present the culmination of the collective efforts of the editors, authors, and the reviewers herein.

The Promises and Perils of Consensus

The image of the big tent is rich. It may conjure up a sweaty crowd of diehard believers who frenetically shout a common response to a leader's call, or it may invoke the obedient who mechanistically follow unspoken rules and policed compliance. Similarly, the image may call to mind a blissful gathering of devotees who unquestioningly accept a philosopher's dogma. On the other hand, the big tent might represent common ground for political allies, a meeting place for friends and constituents, or a stage for fanfare, bragging, and showmanship. We follow Denzin (2008) and expand Tracy's (2010) subsequent use of the metaphor of the big tent as a place to house dialog across paradigms as we consider quality. We are interested in both the perils and the potentials of the big tent as pertains to coherence and divergence around quality, peer reviewing, and qualitative research dissemination. In this volume, we aim to avoid the perils of narrowly defined checklists of quality and what counts as educational research. Toward this end we strive to promote a discussion of quality that avoids mechanistic application of such checklists for the sake of purity, instead focusing on the value of the research endeavor. We aim to highlight the potential of qualitative research to innovatively and creatively address the most enduring challenges and questions in the social sciences.

Quality, as it has been conceptualized in the broad field of qualitative research, is neither easy to articulate nor commonly defined among researchers who engage a variety of methods and methodologies. The breadth of qualitative epistemological, ontological, and methodological approaches is a strength in that such diversity allows for creative and innovative strategies for addressing persistent problems in the social sciences (Lather, 2006). Nevertheless, the diverse philosophies and methodologies that inform researchers’ qualitative research decisions share a common ancestry. Across methods, shared concerns about quality focus on validity, sample selection, participant and researcher relationships and positions, and the ecological frame of qualitative projects.

Coming to shared understandings about quality, as we argue, is important to redress the narrow conceptualizations of quality, evidence, and validity currently privileged in research (Chatterjui, 2008; Freeman et al., 2007). Additionally, we agree with Morrow (2005) that broad quality indicators (e.g., congruency between question and method, and ethical consideration of participants) span quantitative and qualitative methods and that any discussion of quality must be decipherable to all researchers. We also assert, however, that it is the work of qualitative researchers to define quality across internal epistemological, ontological, and methodological approaches from within the field rather than from a prescribed set of axioms that govern quantitative designs. Lincoln (1995) underscored this point in her discussion of “relational” indicators of quality that are outside of the realm of more “traditional inquiry, which posits a detached observer's distance” (p. 278). Articulating shared conceptualizations of quality, we believe, has the potential to initiate dialog among researchers who see themselves as insiders and experts, as well as researchers who see themselves as guests in the field, asked to review journal submissions and grant proposals for their expertise in content, but strangers to the logic of the methods and methodologies under scrutiny. Having a tent under which to gather, then, allows us to locate our work and each other. Ceglowski and colleagues (2011) have highlighted a type of censorship that results when reviewers perpetuate a master narrative in research through the application of narrow definitions of quality criteria that do not align with the methodology and the underlying epistemology and ontology of the inquiry under review. We put forth this guide as a tool for supporting those who wish to participate in the review of qualitative research, not as a tool designed to cut or exclude from participation those who do not comply with some set of arbitrary and predetermined rules. That said, we have encouraged chapter authors throughout the editing process to make claims about quality but to do so in contextualized ways that allow room for reviewers to understand the breadth and scope of the lack of consensus with which the field of qualitative research grapples and, indeed, invites and enjoys.

Our Intentions for this Reviewers’ Guide

It is impossible for all researchers to have expertise in, or in some cases even familiarity with, a wide span of methods. The appeal of the Reviewer's Guide to Quantitative Methods edited by Hancock and Mueller (2010) caught our attention as an excellent resource for seeking additional information when we were unsure of how to judge the merit of a study whose researchers employed an approach with which we were unfamiliar. Coming to the idea that such a publication might be useful for those reviewing qualitative methodologies was neither obvious nor swift. Quantitative research, though diverse, shares common set of assumptions, tools for inquiry, and goals. These shared beliefs and practices promote coherence and adherence to rules, with debate relegated to the most esoteric methods. Clearly, a volume addressing qualitative research and quality would take on a different form. We questioned, as our proposal reviewers also did later, whether such a volume could be useful and we set about building a rationale for this project. We concluded that devoting consistent attention to the foundational aspects of qualitative research would allow the identification of criteria for quality.

We have multiple purposes for this edited volume. First, we have attempted to create a space for a discussion of quality indicators in qualitative research. This goal is not born of a discomfort for flexibility, variation in application, critique, or innovation in research. In fact, we see these as defining characteristics of qualitative research, as those from which great potential emerges in the construction of transformative knowledge. We understand the reluctance to endorse our endeavor by some scholars (with whom we sometimes agree) that benchmarks or guides can be overly prescriptive or simplistic when applied routinely without contextualization. Caution is prudent; in our own fields within educational research, ongoing efforts to define quality indicators of rigorous research have been narrowly interpreted and applied to the exclusion of funding and dissemination of qualitative research that strays afield from a postpositivist epistemology and a realist ontology.

Intere...