Children's Reading Comprehension and Assessment

Scott G. Paris, Steven A. Stahl

• Publisher: Routledge
• Year: 2005
• eBook ISBN: 9781135621612
• Topic: Education
• Subtopic: Education General

Originating in a recent CIERA conference held at the University of Michigan, this book brings together the nation's most distinguished researchers to examine how readers understand text and how comprehension is assessed. The first part provides both national and historical contexts for the study of reading comprehension. The second part examines how vocabulary, motivation, and expertise influence comprehension, and it includes analyses of the developmental course and correlates of comprehension. Chapters in the third part consider how schools focus on comprehension for instruction and assessment. The fourth part includes chapters on large-scale assessment that analyze how test formats and psychometric characteristics influence measures of reading comprehension. At the end of each part is a commentary--written by an expert--that reviews the chapters, critiques the main points, and synthesizes critical issues.

Key features of this outstanding new book include:

*Integration of Research and Practice--provides a bridge between conceptual issues studied by researchers concerned with reading comprehension theories and practical issues addressed by educators concerned with classroom instruction and assessment.

*Comprehension Focus--provides a thorough history and rigorous research-based analyses of reading comprehension.

*Assessment Focus--provides innovative approaches to comprehension assessment that include the influences of vocabulary, decoding, and motivation.

*Synthetic Commentaries--provides periodic summaries that analyze and synthesize research, practices, and issues discussed in each part.

*Expertise--contributing authors and commentators are highly respected authorities on reading comprehension (see table of contents).

This text is appropriate for educational and psychological researchers, reading educators, and graduate students in education and psychology. It is part of the CIERA series, which includes the following volumes:

Taylor and Pearson: Teaching Reading: Effective Schools, Accomplished Teachers (2002)

Van Kleeck, Stahl, and Bauer: On Reading Books to Children: Parents and Teachers (2003)

Hoffman and Schallert: The Texts in Elementary Classrooms (2005)

Part I Historical and Theoretical Foundations

1 Assessment of Reading Comprehension: The RAND Reading Study Group Vision

Anne P.Sweet
U.S. Department of Education

The assessment of reading comprehension is a critical component of any national research effort aimed at improving our understandings about what is reading comprehension and how it can best be taught. The core of this chapter, excerpted from Reading for Understanding: Toward an R&D Program in Reading Comprehension (2002), was written by the RAND Reading Study Groups for the Department of Education’s Office of Educational Research and Improvement (OERI) and designed to inform the development of a research agenda for reading. In 1999, Kent McGuire, then assistant secretary for OERI, launched an agendasetting effort for federal education research, focused on mathematics and reading education and managed by the RAND Corporation. Two study groups were formed, each charged with identifying the most pressing needs in its particular area.

The RAND Reading Study Group (RRSG) was composed of 14 experts,¹ representing a range of disciplinary and methodological perspectives on the field of reading. This group functioned as an expert panel for little more than 2 years (2000–2002) to establish a convergent perspective on what is known about reading, what are the most urgent tasks in developing an integrated research base, and what needs to be done to improve reading outcomes. The study group formulated an initial draft of a report in the summer of 2000 which was used to solicit commentary and guidance to the committee in devising its final report. That report was published early in 2002 as a book entitled Reading for Understanding: Toward an R&D Program in Reading Comprehension (RAND Reading Study Group, 2002). The RRSG report served its primary purpose by providing the impetus for OERI to create a whole new Program of Research on Reading Comprehension (PRRC), under the leadership of successor OERI assistant secretary, Grover J. Whitehurst.

We have made enormous progress over the last 25 years in understanding how to teach aspects of reading. We know about the role of phonological awareness in cracking the alphabetic code, the value of explicit instruction in teaching sound-letter relations, and the importance of reading practice in creating fluency. Measures of these constructs (phonological awareness, letter-sound relations, and fluency), on average, render good approximations of these skills. The same cannot be said for measures of reading comprehension.

THE STUDY GROUP’S ANALYSIS

Understanding the nature of the problem of reading comprehension requires having available good data identifying which readers can successfully undertake which activities with which texts. Such data are not available, in part because the widely used comprehension assessments are inadequate. Further, the improvement of instruction relies crucially on the availability of information about the effectiveness of instruction. Teachers need reliable and valid assessments tied closely to their curricula so that they can see which students are learning as expected and which need extra help. In addition, schools, districts, and states are increasingly calling for reliable and valid assessments that reflect progress toward general benchmarks of reading, writing, and mathematics ability. For the area of reading comprehension, good assessments that are tied to curriculum as well as good assessments of general comprehension capacity are sorely needed. These assessments need to be constructed in accordance with the many advances in psychometric theory.

WHAT WE ALREADY KNOW ABOUT COMPREHENSION ASSESSMENTS

Currently available assessments in the field of reading comprehension generate persistent complaints that these instruments

• inadequately represent the complexity of the target domain.
• conflate comprehension with vocabulary, domain-specific knowledge, word reading ability, and other reader capacities involved in comprehension.
• do not rest on an understanding of reading comprehension as a developmental process or as a product of instruction.
• do not examine the assumptions underlying the relation of successful performance to the dominant group’s interests and values.
• are not useful for teachers.
• tend to narrow the curriculum.
• are unidimensional and method-dependent, often failing to address even minimal criteria for reliability and validity.

Indeed, most currently used comprehension assessments reflect the purpose for which they were originally developed—to sort children on a single dimension by using a single method. Even more important, however, is that none of the currently available comprehension assessments is based in a viable or articulated theory of comprehension. In addition, none can give us a detailed or convincing picture of how serious is the problem of comprehension achievement in the United States. These considerations, as well as the thinking about the nature of reading comprehension represented in this document, create a demand for new kinds of assessment strategies and instruments that (a) more robustly reflect the dynamic, developmental nature of comprehension; (b) represent adequately the interactions among the dimensions of reader, activity, text, and context; and (c) satisfy criteria set forth in psychometric theory.

Currently, widely used comprehension assessments are heavily focused on only a few tasks: reading for immediate recall, reading for the gist of the meaning, and reading to infer or disambiguate word meaning. Assessment procedures to evaluate learners’ capacities to modify old or build new knowledge structures, to use information acquired while reading to solve a problem, to evaluate texts on particular criteria, or to become absorbed in reading and develop affective or aesthetic responses to text, have occasionally been developed for particular research programs but have not influenced standard assessment practices. Because knowledge, application, and engagement are the crucial consequences of reading with comprehension, assessments that reflect all three are needed. Further, the absence of attention to these consequences in widely used reading assessments diminishes the emphasis on them in instructional practices as well.

WHAT WE NEED IN THE ARE A OF COMPREHENSION ASSESSMENTS

The entire research enterprise sketched out in this report depends on having a more adequate system of instrumentation for assessing reading comprehension. A satisfactory assessment system is a prerequisite to making progress with all aspects of the research agenda we propose. Thus we argue that investing in improved assessments has very high priority, It is clear that we cannot even sketch the seriousness of the problem of reading comprehension in the United States or the nature of the decline in comprehension outcomes that is the source of much worry until we have an assessment system that can be used across the developmental range of interest and that assesses the same construct across that range.

Assessing the effect of changes in instruction depends on having valid, reliable, and sensitive assessments. The effect of assessment on instruction is a question that constitutes a research agenda of its own, particularly in this highly accountability-oriented era of education reform. However, the power of high-stakes assessments over instruction and curriculum can be somewhat mitigated if teachers have available alternative assessment options that give them more useful information.

Any system of reading assessments should reflect the full array of important reading comprehension consequences. We argue that a research program to establish expectable levels of performance for children of different ages and grades on this full array of consequences is necessary. Such a program is a prerequisite to developing performance criteria at different age and grade levels and to pursuing questions about reader differences associated with instructional histories, social class, language, and culture in reading comprehension outcomes.

Although the reading comprehension consequences defined earlier constitute the basis for designing a comprehension assessment that would reflect success, our view suggests that assessments designed to reflect readers’ cognitive, motivational, and linguistic resources as they approach a reading activity are also necessary. For instance, when the outcomes assessment identifies children who are performing below par, process assessments could help indicate why their reading comprehension is poor. Further, diagnostic assessments are crucial in dissecting the effect of particular instructional or intervention practices. Ideally, we would move ultimately toward assessment systems that can also reflect the dynamic nature of comprehension, for example, by assessing increments of knowledge about vocabulary and particular target domains that result from interaction with particular texts.

We see the development of an assessment system for reading comprehension as having a very high priority. Such a system should be based in contemporary approaches to test development and evaluation. We recognize that developing a comprehensive, reliable, and valid assessment system is a long-term project. Crucial for such a system are the criteria for judging performance across the developmental span. Nonetheless, a substantial start could be made in the short run, either by targeting the assessment of outcomes and reader resources as a major task of the research agenda or by encouraging the development of prototype assessments for outcomes and reader resources within other research efforts (such as research focused on instructional efficacy), Such an effort is central to pursuing larger research agendas, such as longitudinal work to create a picture of the development of reading comprehension, a largescale effort to determine how U.S. children are functioning as readers, or a systematic pursuit of differences in reading comprehension performance related to cultural background, social class, and language status.

The approach to assessment proposed here differs from current approaches to reading assessment in that it would both grow out of and contribute to the development of an appropriately rich and elaborated theory of reading comprehension. Assessment procedures generated by this approach are thus more likely to be influenced and changed by theoretically grounded reading research. Our approach also highly values the utility of assessment for instruction. Of course, comprehensive assessment systems can place high demands of time on students and teachers; thus, we have an obligation to develop assessments that are embedded in and supportive of instruction, rather than limited to serving the needs of researchers.

A comprehensive assessment program reflecting the thinking about reading comprehension presented here would have to satisfy many requirements that have not been addressed by any assessment instruments, while also satisfying the standard psychometric criteria. The minimum requirements for such a system follow:

• Capacity to reflect authentic outcomes—Although any particular assessment may not reflect the full array of consequences, the inclusion of a wider array than that currently being tested is crucial. For example, students’ beliefs about reading and about themselves as readers may support or obstruct their optimal development as comprehenders; teachers may benefit enormously from having ways to elicit and assess such beliefs.
• Congruence between assessments and the processes involved in comprehension—Assessments that target particular operations involved in comprehension must be available, in the interest of revealing interindividual and intraindividual differences that might inform our understanding of the comprehension process and of outcome differences. The dimensionality of the instruments in relation to theory should be clearly apparent.
• Developmental sensitivity—Any assessment system needs to be sensitive across the full developmental range of interest and to reflect developmentally central phenomena related to comprehension. Assessments of young children’s reading tend to focus on word reading rather than on comprehension. Assessments of listening comprehension and of oral language production, both of which are highly related to reading comprehension, are rare and tend not to be included in reading assessment systems despite their clear relevance. The available listening comprehension assessments for young children do not reflect children’s rich oral languageprocessing capacities because they reflect neither the full complexity of their sentence processing nor the domain of discourse skills.
• Capacity to identify individual children as poor comprehenders— An effective assessment system should be able to identify individual children as poor comprehenders, not only in terms of prerequisite skills such as fluency in word identification and decoding, but also in terms of cognitive deficits and gaps in relevant knowledge (background, domain specific, etc.) that might adversely affect reading and comprehension, even in children who have adequate word-level skills. It is also critically important that such a system be able to identify early any child who is apt to encounter difficulties in reading comprehension because of limited resources to carry out one or another operation involved in comprehension.
• Capacity to identify subtypes of poor comprehenders—Reading comprehension is complexly determined. It therefore follows that comprehension difficulties could come about because of deficiencies in one or another of the components of comprehension specified in the model. Thus, an effective assessment system should be able to identify subtypes of poor comprehenders in terms of the components and desired outcomes of comprehension. It should also be capable of identifying both intraindividual and interindividual differences in acquiring the knowledge and skills necessary for becoming a good comprehender.
• Capacity to identify subtypes of poor comprehenders—Reading comprehension is complexly determined. It therefore follows that comprehension difficulties could come about because of deficiencies in one or another of the components of comprehension specified in the model. Thus, an effective assessment system should be able to identify subtypes of poor comprehenders in terms of the components and desired outcomes of comprehension. It should also be capable of identifying both intraindividual and interindividual differences in acquiring the knowledge and skills necessary for becoming a good comprehender.
• Instructional sensitivity—Two major purposes for assessments are to inform instruction and to reflect the effect of instruction or intervention. Thus, an effective assessment system should provide not only important information about a child’s relative standing in appropriate normative populations (school, state, and national norms groups), but also important information about the child’s relative strengths and weaknesses for purposes of educational planning.
• Openness to intraindividual differences—Understanding the performance of an individual often requires attending to differences in performance across activities with varying purposes and with a variety of texts and text types.
• Usefulness for instructional decision making—Assessments can inform instructional practice if they are designed to identify domains that instruction might target, rather than to provide summary scores useful only for comparison with other learners’ scores. Another aspect of utility for instructional decision making is the transparency of the information provided by the test given to teachers without technical training.
• Adaptability with respect to individual, social, linguistic, and cultural variation—Good tests of reading comprehension, of listening comprehension, and of oral language production target authentic outcomes and reflect key component processes. If performance on a task reflects differences owing to individual, social, linguistic, or cultural variations that are not directly related to reading comprehension performance, the tests are inadequate for the purposes of the research agenda we propose here.
• A basis in measurement theory and psychometrics—This basis should address reliability within scales and over time, as well as multiple components of validity at the item level, concurrently with other measures and predictively relative to the longer-term development of reading proficiency. Studies of the dimensionality of the instruments in relation to the theory underpinning their construction are particularly important. Test construction and evaluation of instruments are important areas of investigation and are highly relevant to our proposed research agenda.

Clearly, no single assessment would meet all these criteria. Instead, we propose an integrated system of assessments, s...

Table of contents

• COVER PAGE
• TITLE PAGE
• COPYRIGHT PAGE
• DEDICATION
• FOREWORD
• PREFACE
• PART I HISTORICAL AND THEORETICAL FOUNDATIONS
• PART II DEVELOPMENTAL AND MOTIVATIONAL FACTORS IN READING COMPREHENSION
• PART III ASSESSMENT IN SCHOOL CONTEXTS
• PART IV LARGE-SCALE ASSESSMENTS OF READING COMPREHENSION