The Critical Assessment of Research

Traditional and New Methods of Evaluation

Alan Bailin, Ann Grafstein

• Publisher: Chandos Publishing
• Year: 2010
• Print ISBN: 9781843345442
• eBook ISBN: 9781780630274
• Topic: Education
• Subtopic: Education General

This book examines the following factors: sponsorship of research, control of the dissemination of research, effects of dominant research paradigms, financial interests of authors, publishers, and editors, role of new technologies (for example, Web 2.0).It is widely accepted among researchers and educators that the peer review process, the reputation of the publisher and examination of the author's credentials are the gold standards for assessing the quality of research and information. However, the traditional gold standards are not sufficient, and the effective evaluation of information requires the consideration of additional factors. Controversies about positive evaluations of new medications that appear in peer-reviewed journals, the financial reports on Enron prior to the revelations that led to its collapse, and obstacles to the publication of research that does not conform to dominant paradigms are just a few examples that indicate the need for a more sophisticated and nuanced approach to evaluating information.Each of the factors is discussed in a factual manner, supported by many examples that illustrate not only the nature of the issues but also their complexity. Practical suggestions for the evaluation of information are an integral part of the text.- Highlights frequently overlooked criteria for evaluating research- Challenges the assumption that the gold standards for evaluation are sufficient- Examines the role of new technologies in evaluating and disseminating research

1

Introduction

Research is an integral part of our world. It is responsible for, among other things, the medicines that we take, our economic policies, our approaches to marketing, the educational strategies used in our schools, therapeutic strategies for the mentally distressed and the techniques for harnessing energy for industry. This book is about evaluating research. It is about assessing studies that have shaped our lives in fundamental ways.

Research is to a great extent the domain of experts and specialists: scientists hired by governments and industries, professors in universities, PhDs working for foundations or in research centers. This book, however, is not for them, at least not in so far as they are experts. It does not discuss the fine points of evaluating the internal consistency of theories, nor the empirical coverage of different kinds of theoretical models. As important as all of these discussions are for the specialists engaged in research, they are not within the domain of this book.

This book is for the rest of us: those of us who read about research but are not experts in the field. It is for those of us who search for ways of understanding important ideas and try to identify poorly thought-out proposals in areas in which we have some interest or concerns, but in which we have not been trained to be specialists. Since even experts are only experts in limited areas, this means that this book is for all of us.

All of us need ways to assess the research findings we encounter. Whether we are reading about the newest medical discoveries, trying to decide where to invest our money, considering the proposals of politicians for the development of new energy resources – in all of those many areas in which research and research findings affect our lives and we have no way of making an expert assessment – all of us need a basic knowledge of what to look for, and what to look out for.

When research is reported in popular media outlets, its findings are often presented as established facts, and even sometimes as a clarion call for immediate action. Nevertheless, it sometimes turns out that these calls to action are based on questionable research. In the 1990s, for example, menopausal women were urged by such reputable sources as the American Heart Association (Felgran and Hettinger, 2002: 71) and the American College of Physicians (Kolata and Petersen, 2002) to have hormone replacement therapy (HRT) in order to decrease the chances of developing heart disease and osteoporosis. By 2002 HRT was viewed far more skeptically as a result of later findings: while it indeed lowered the risk of osteoporosis, at the same time it increased the risk of stroke, heart disease, breast cancer and dementia (Writing Group for the Women’s Health Initiative Investigators, 2002).

The financial sphere is another example of an area where information, even when disseminated from reputable sources, requires close scrutiny. Enron was considered one of the fastest-growing companies and a safe investment. Anyone researching the company found reassuring financial statements based on apparently impeccable accounting research. Nevertheless, the collapse of Enron became worldwide news. It might be tempting to think that the Enron situation is an aberration, but the accounting conflicts of interest that contributed to it have been widespread (Fearnley et al., 2005: 69).

The intention of the book is to present the reader with some of the basic concepts that can be used for assessing the quality of research, regardless of its topic or area of specialization. It tries to point the reader to warning flags that ought to prompt further questions and perhaps some skepticism. In sum, this book is about assessing research as a non-expert in an intelligent, educated manner.

There are tried-and-true methods of deciding whether particular research is worth even considering. In the second chapter we discuss these criteria, showing why they are used and pointing out where they can be useful. However, we also show you where these criteria fall short, and why trusting these gold standards as the primary criteria for assessing research can lead to misplaced confidence.

In Chapters 3–5 we look at issues that can affect the validity of research. In particular, we look at questions related to the funding of research, the theoretical models upon which the research is based and the venues available for disseminating it. In each case we investigate the way in which economic and ideological systems can affect research and its outcomes.

In the third chapter we examine the issues of funding and sponsorship. We show how the sources that fund and sponsor research may affect its outcomes and conclusions. In so doing, we look at three different cases in quite different fields. First we examine the ways in which pharmaceutical funding and sponsorship of drug-related research have affected the way in which the findings of that research are viewed. We look at how pharmaceutical companies used funding to spin the research evidence to make hormone replacement therapies appear safer than the research might have otherwise suggested.

Next, we look at the Enron debacle to see how funding conflicts of interest can corrupt the investigations of auditors (in other words, their research) into a company’s financial health. We discuss how funding played a decisive role in corrupting the auditing of Enron’s financial status and thus allowed Enron to produce flawed financial reports – reports that were nevertheless thought to be sound because the auditors had investigated Enron’s reporting practices and approved them.

We then move to the field of psychology and the appearance of a much-heralded book, The Bell Curve, which claimed to establish racial differences in IQ scores. Although it might seem that the funding of research would relate solely to economic self-interest, The Bell Curve illustrates how funding can be used to promote ideological agendas as well. We discuss how one of the authors was supported by foundations that have specific ideological agendas. We address the convergence between the agendas of the funding sources and the findings of the research. Our interest is not in whether or not the book’s conclusions are ‘true’ or offensive, but rather whether or not this convergence should prompt questions about the research.

While financial conflicts of interest are the stuff of which magazine and newspaper articles are often made, there are other considerations which are far subtler but can nevertheless affect the nature of research. In the fourth chapter we look at research from the perspective of how dominant research models (paradigms) exert influence not only over the conclusions of research studies, but even over the questions that are asked, the hypotheses that are investigated and the subjects that are studied. Theoretical paradigms constitute the set of assumptions – that is ideas – that form the framework within which research takes place. The focus of the fourth chapter is thus on the relationship between ideological systems and research.

We look first at one of the most influential research applications around the world, IQ tests, and show that the research supporting the use of these tests and the claim that they identify inheritable traits depends on particular theoretical assumptions. Different assumptions, we suggest, lead to different conclusions about intelligence and the meaning of its inheritability. We argue that an awareness of alternative perspectives is important in critically evaluating this research.

The next case we look at involves the causes of ulcers. Research throughout much of the twentieth century was based on a theoretical model that emphasized the role of stress in the development of ulcers. Treatment regimens were routinely recommended on the basis of this assumption. But late in the twentieth century a radically different theoretical model of ulcers was proposed. This model met with considerable resistance because it differed substantially from the dominant theoretical paradigm. Nevertheless, the newer model eventually gained acceptance because it led to more effective treatments. In science, accepted paradigms are periodically overthrown and replaced. In critically assessing research, we suggest, it is prudent to keep in mind that theories that seem far-fetched today may become the established truth of tomorrow.

Science is hardly the only area in which dominant paradigms affect research. In our final case study in this chapter we look at the effect of canons on research topics in the humanities and fine arts. We show how for years the focus of the Anglo-American literary canon meant that little research investigated anyone but British male writers. In recent years academicians have discovered, however, that there was serious literature created by a far more diverse group of writers, whose work had been ignored because it was not part of the canon. A similar trend can be found in the area of fine arts, and we discuss this with particular reference to female artists. We discuss how the canon discourages research into subjects that are excluded from it.

In Chapter 5 we turn to the dissemination of research. In the first case study we show that since the second half of the twentieth century there has been an explosion of journals created by groups that did not view themselves as adequately represented by mainstream journals, frequently because of ideological reasons. The fields of women’s studies and gay studies developed to address the intellectual needs of groups that were struggling for power. Academic journals were started in order to publish and disseminate research on topics of specific interest to these emerging fields of study. We point out that when assessing research in these areas, it is important to consider research from these newer publication venues. Considering publications that offer new and different perspectives on topics, we suggest, provides a more complete context for critically assessing research.

In the next case study we focus on the effects of economic rather than ideological interests. We consider instances in which pharmaceutical companies sponsoring research have influenced its dissemination. We show that sponsorship has sometimes led to the cherry-picking of the data that are reported and even the outright suppression of research findings.

Finally, we look at the ways in which governments, foundations, corporations and other institutions support and disseminate research in what is known as ‘gray literature’. We show that gray literature often fills the gaps left by traditional publication venues. These gaps may be the result of either economics or ideology; in either case gray literature provides access to research that may not be disseminated in other ways. As with the new journals, gray literature can thus provide a perspective that would otherwise be missing when we critically assess research.

In the sixth and final chapter we discuss practical ways to go beyond the gold standards for evaluating research and find the information you need to make sound assessments. We suggest ways to track down the funding sources of research and find out about alternative publication venues and different research paradigms, using both print and digital resources. Much of this information is available for free on the web and in libraries, and we show you how to use these resources efficiently and effectively. We conclude by discussing how to strike the appropriate balance in the critical assessment of research, viewing research with a skeptical eye without rejecting the important information and perspectives it can provide.

2

The gold standards

Introduction

If you are not an expert in a field of research, how are you going to evaluate the reliability of research findings? As we said in the Introduction, this is not an abstract or theoretical question, of importance only to scientists and other scholars who care about research for a living. The mass media (television, radio, newspapers, magazines and, more recently, internet sites) disseminate major research findings, since they are of interest to and have significant implications for the general public. We base many of our most important decisions on reports about research. Sometimes this research gives us good information and sometimes it does not. Sometimes what research seems to tell us we find out later is just plain wrong, even dangerous.

In the first chapter we mentioned research about hormone replacement therapy (HRT), and pointed out that in the 1990s medical research seemed to suggest that HRT could help prevent heart disease and osteoporosis. HRT was routinely prescribed for healthy menopausal women, based on the assumption that it would protect them from both osteoporosis and heart disease. However, by 2002 further research suggested that this apparently effective treatment was not nearly as safe as had been previously thought. HRT was found to be implicated in heart disease, breast cancer and even dementia (Palmlund, 2006).

What are people who are not experts to make of such reversals? Research indicates a certain course of action, expert practitioners – in this case medical doctors – act on this research, and then it turns out that the research was at best misleading if not simply wrong. How can an intelligent, educated person make sense of it all? Is it best to trust absolutely nothing? Alternatively, should we simply have faith in the research and hope all works out for the best?

Peer review

Clearly we need to be able to assess research critically. In many, perhaps most, cases, all we have to go on is what we read or hear about research, and often what we read or hear is second-hand information reported in the mass media. However, although at times researchers make themselves available for interviews and sometimes even give press conferences, almost all research is only fully presented in specialized peer-reviewed journals or books intended for experts in the field.

The key term here is ‘peer review’, and it is perhaps the most important of the gold-standard criteria for evaluating research. It is essential to understand what ‘peer review’ means, the pivotal role it plays in the dissemination of research and its significance as a tool for assessment by non-experts.

Peer review is basically a selection process. When scientists or scholars have research findi...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• About the authors
• Acknowledgements
• Chapter 1: Introduction
• Chapter 2: The gold standards
• Chapter 3: Sponsorship and funding
• Chapter 4: Research paradigms
• Chapter 5: The dissemination of research
• Chapter 6: Moving beyond the gold standards: tools and techniques
• References
• Index