A Multidimensional Approach to Achievement Validation

A Special Issue of Educational Assessment

Richard J. Shavelson, Robert W. Roeser, Richard J. Shavelson, Robert W. Roeser

• Publisher: Routledge
• Year: 2018
• eBook ISBN:
  9781135065881
• Topic: Education
• Subtopic: Education General

Two of the most interesting conceptual turns in Richard E. Snow's thinking called for: a broadening of the concept of aptitude to include not only cognitive processes, but also affective and cognative processes as essential for understanding academic performance and learning; and an exploration of the possibility that individual differences in learning and achievement emerge from dynamic person-situation transactions that unfold over time. The articles in this special issue address these "big ideas" through the lens of a study of high school students' achievement in science.

Richard E. Snow's Remaking of the Concept of Aptitude and Multidimensional Test Validity: Introduction to the Special Issue

Richard J. Shavelson
School of Education Stanford University/CRESST

Haggai Kupermintz
School of Education University of Colorado at Boulder/CRESST

Carlos Ayala
School of Education Stanford University/CRESST

Robert W. Roeser
School of Education Stanford University

Shun Lau
School of Education Stanford University

Angela Haydel
School of Education Stanford University

Susan Schultz
School of Education Stanford University/CRESST

Larry Gallagher and Gisell Quihuis
School of Education Stanford University

Richard E. Snow conceived of individuals' performances as the result of a transaction between their aptitudes and the particular characteristics of the situation in which a performance occurred over time. By aptitudes he meant the cognitive, conative, and affective resources that an individual brings to the situation. By situation he meant the characteristics of a particular environment (e.g., test) that afforded or impeded—that assisted or constrained—certain courses of goal-directed action for different individuals. In relation to achievement tests. Snow proposed that individuals' test performances emerged from the organization of their demographic background and intellectual history; an "assembly" of cognitive, conative, and affective processes needed to more or less effectively respond to a series of situation-embedded test tasks (e.g., multiple-choice items); and the ways in which the demand characteristics and opportunities of the tasks were attuned to or mismatched with individuals' aptitude resources. Performance and its determinants were not viewed as static but rather as dynamic processes unfolding over time at the interface of person and situation. From this reasoning and related empirical findings. Snow concluded that a new multivariate approach to validating interpretations of achievement test scores was needed. In this introduction to this special issue, we describe some of Snow's "big ideas" regarding issues of aptitude, person-situation transactions, and test validity. We then describe the design of the "high school study" we undertook, after Snow's passing in 1997, to explore some of these ideas further.

In 1995, Richard E. Snow wrote in a proposal to the Office of Educational Research and Improvement that his previous work (e.g., Hamilton, Nussbaum, & Snow, 1997; Nussbaum, Hamilton, & Snow, 1997) showed that "psychologically meaningful and useful subscores can be obtained from conventional achievement tests" (Snow, 1995, p. 15; see also Baker, Linn, & Herman, 1995, p. 133). He went on to point out that these subscores represented important ability distinctions and showed different patterns of relationships with demographic, "affection" (temperamental-emotional), "conative" (motivational-volitional), and instructional-experience characteristics of students. He concluded that " a new multidimensional approach to achievement test validation should include affective and conative as well as cognitive reference constructs" (Snow, 1995, pp. 16-17; see also Baker et al., 1995, p. 134). Unfortunately, he was unable to carry out this research himself due to his untimely death in 1997. Thus, a group of researchers and graduate students (this article's authors) came together and completed the high school study to carry out some of Snow's ongoing work. We begin this introduction by expanding on Snow's ideas, building a framework for conceiving multidimensional validity. Next, we describe the design of the high school study.

"Big Ideas" and a Metatheoretical Umbrella

The Remaking of the Concept of Aptitude

In perhaps two of the most important conceptual turns in his thinking, Snow called for (a) a broadening of the concept of aptitude to include not only cognitive processes (e.g., abilities, strategies, knowledge) but also affective and conative processes as essential for understanding academic performance and learning; and (b) an exploration of the possibility that individual differences in learning and achievement emerged from dynamic person-situation transactions that unfolded over time (Snow, 1989, 1994; see also Corn et al., 2002). In the first instance, Snow advocated for a "whole person" approach to conceptualizing and researching the full spectrum of psychological inputs to academic performance and learning. These processes included those manifold motivational and self-regulatory processes, necessary for bringing the individual to an intellectual task and for energizing and sustaining goal-directed behavior in relation to the task, as well as those manifold cognitive processes necessary for the individual, once engaged, to successfully perform the operations necessary to solve the intellectual problems posed by the task. By aptitudes, Snow meant all these characteristics (e.g., experience, ability, knowledge, motivation, and regulatory processes) that an individual brings to and cobbles together to perform in a particular situa...

Table of contents

• Cover
• Contents
• Richard E. Snow's Remaking of the Concept of Aptitude and Multidimensional Test Validity: Introduction to the Special Issue
• Reasoning Dimensions Underlying Science Achievement: The Case of Performance Assessment
• Affective and Conative Factors as Aptitude Resources in High School Science Achievement
• Cognitive Abilities and Motivational Processes in High School Students' Situational Engagement and Achievement in Science
• On Motivation, Ability, and the Perceived Situation in Science Test Performance: A Person-Centered Approach With High School Students
• The Concept of Aptitude and Multidimensional Validity Revisited