Evaluating Research Articles From Start to Finish

Ellen Robinson Girden, Robert Ira Kabacoff

• Publisher: SAGE Publications, Inc
• Year: 2010
• ISBN:
  9781483343204
• Edition: 3
• Topic: Didattica
• Subtopic: Ricerca nella didattica

This thoroughly updated new edition of the bestselling text trains students—potential researchers and consumers of research—to critically read a research article from start to finish. Containing 25 engaging samples of ideal and flawed research, the text helps students assess the soundness of the design and appropriateness of the statistical analyses.

Chapter 1

---

Introduction

---

The purpose of this book is to train students—potential researchers and consumers of research—to critically read a research article from start to finish. You will learn to critically read an introduction, which sets the stage by describing the rationale for the study (i.e., what led to it) as well as its purpose (i.e., what the study hoped to accomplish). You will learn how to “dissect” the method section so that you can decide whether precautions were taken to guard against threats to internal validity, both in terms of assignment of participants to the various conditions of the study and in use of control procedures and groups. You will become more familiar with interpreting results and even with performing additional calculations or checking a particular result. Finally, you will learn to carefully evaluate the experimenter’s discussion of the results to determine the extent to which the conclusion is justified, can be generalized, and has limitations.

Studies are presented in order of increasing complexity. Each is prefaced with an introduction that describes the basic design and statistical analysis. Every effort has been made to locate examples of good as well as flawed studies in each of the categories, ones that performed statistical analyses that are commonly taught at intermediate and advanced levels. When it is feasible, articles are presented verbatim. For the most part, however, sections have been excerpted and/or revised for clarity. Ellipses and bracketed phrases are used to indicate some changes. More extensive revisions or synopses are indented from the margins, enclosed in brackets, and always appear in italics.

All of the chapters (with the exception of Chapter 3) contain two examples of studies that employed a particular design. You will begin by evaluating the first study with our assistance. Copious notes are added to alert you about potential flaws or positive aspects of the design. These are indicated by an arrow in the top left; they are enclosed in parentheses and always appear in italics. When we evaluate articles together, questions and answers are shaded to make them distinct. The second article is excerpted, and it is followed by a series of critical guide questions. Following questions, a page number directs you to the answers, which are found in a separate section at the end of the book.

The remainder of this chapter presents a review of the bases of empirical research (controlled observations that are reliable and valid) followed by a review of potential trouble spots that can invalidate a conclusion about the effectiveness of an independent variable.

---

A SCIENTIFIC APPROACH

---

Before discussing the nuts and bolts of empirical research, let’s discuss the scientific endeavor more generally, in order to place the methods discussed in this book within a useful context. Much of the philosophy of science is concerned with the nature of cause and effect. Philosophers from a wide range of traditions (positivist, essentialist, activity theorists, and evolutionary critical realists) have all been concerned with the following question: “How can we determine the cause of an event?”

Research in the social sciences has been strongly influenced by the writings of John Stuart Mill and Karl Popper. Mill held that causal inference requires that (a) cause has to precede effect in time, (b) cause and effect must be related, and (c) all other explanations for the relationship must be eliminated. If the presumed cause is present whenever the effect is present (method of agreement), the effect is absent whenever the cause is absent (method of difference), and both can be observed repeatedly (method of concomitant variation), we have evidence for causality. For Popper, the most important feature of the scientific approach involved the falsification (rather than the confirmation) of theories. Differing explanations (theories) for observations are placed in competition with each other. The theory that best explains the data (in terms of simplicity, predictive power, and the ability to incorporate new data) is retained, until replaced by a better theory.

Both Mill and Popper had a healthy appreciation for the need to examine and eliminate alternative explanations for the findings before settling on a presumed cause. In the social sciences, the removal of such alternate explanations (called confounding factors) is a constant struggle, as you will see throughout this book.

The scientific method typically follows the hypothetico-deductive approach outlined here:

1. Make observations about a phenomenon.
2. Form hypotheses (proposed explanations for the observed phenomenon).
3. Make predictions based on these hypotheses.
4. Test the predictions through observation and experimentation.
5. Based on the results, form new hypotheses (Step 2), and repeat Steps 3, 4, and 5 in an iterative fashion.

To the degree that a theory leads to testable predictions, which are confirmed in repeated assessments by multiple scientists in varying settings and cannot be explained by confounding factors or better alternate theories, the greater weight and stature it is given.

---

EMPIRICAL RESEARCH

---

There are three key concepts associated with empirical research: controlled observation, reliability, and validity. Controlled observation refers to the precision of conditions under which data are collected. In essence, any “noise” that can affect the data is eliminated, minimized, or counteracted in such a way that any other observer can replicate the conditions. Superfluous factors in the environment are eliminated or minimized by gathering data under uniform conditions; environmental distractions (e.g., sights or sounds) are the same for all participants, surrounding temperature is the same, and the data collector is the “same” (at the same level of expertise throughout testing)—or if more than one is used, they are equally distributed throughout the various groups and so forth. These features ensure that the collected data will be objective, precise, and verifiable.

Reliability refers to a broad range of phenomena. Reliability means repeatability or consistency. Empirical research should be reliable: Under the same experimental conditions, anyone else should be able to obtain the same results (i.e., the outcome of data collection should lead to the same conclusion). Reliability also refers to precision of our measuring instruments. Precise instruments are more likely to yield consistent measures than cruder instruments. For example, a determination of 4 oz (113.398 g) of liquid will be more reliable if a calibrated measuring cup rather than a drinking glass is used.

Reliability also refers to the extent to which a test measures consistently or yields a “true” or accurate measure of whatever it is the test measures. That accuracy shows up in two ways: the repeatability of the score on more than one occasion and the same relative standing of the individuals in their group on more than one occasion.

Validity is the final key concept of empirical research. It is synonymous with appropriateness, meaningfulness, and usefulness. With regard to research, we want to know whether conclusions are valid; are they appropriate, meaningful, and useful on the basis of the intent of the investigator and the procedures used to fulfill that intent? There are three types of study validity that we must be concerned with: internal validity, statistical conclusion validity, and external validity.

A study has internal validity to the degree that it allows us to conclude that a relationship between variables is causal or that the absence of a relationship implies a lack of cause. Statistical conclusion validity refers to the appropriateness of the statistical methods employed to determine if covariation exists or not. A study has external validity to the degree that the results can be generalized beyond the current study to situations that use other measures, methods, and populations. Our goal in research is to devise studies that allow us to derive clear and unambiguous answers to the questions posed. To do this, we seek to design our research to limit, to the greatest extent possible, the threats posed to each of form of validity.

---

THREATS TO INTERNAL VALIDITY

---

To evaluate the soundness of each design, you need to keep in mind potential sources of confounds (other potential explanations of results). These are variables that may be operating in conjunction with the manipulated independent variable and make it impossible to determine whether observed changes or differences in the dependent variable are due to the manipulation, the confound, or a combination of the two. Because these potential confounds may threaten the extent to which the conclusion is valid or justified (i.e., internal validity of the study), they are called threats to internal validity. Some threats to internal validity apply to study designs that incorporate pretests and posttests. Some threats apply to research designs in general.

Studies With Pretests and Posttests

History. This refers to any event occurring in the interim that directly or indirectly could affect the behavior being measured and therefore also could account for the results.

Initial testing. This refers to a change in posttest performance that results from pretest experience.

Instrumentation. This refers to any change in the measuring instrument and/or assessor from pretest to posttest that can just as easily explain a change in scores.

Maturation. This refers to any change within the participant that occurs during the interim and can just as easily account for posttest performance.

Regression toward the mean. This is a predicted shift in posttest scores when participants were specifically selected because their pretest scores were extremely high or low. Posttest scores are predicted to be less extreme, regardless of treatment ef...