Assessment in Mathematics Education Contexts

Theoretical Frameworks and New Directions

Jonathan D. Bostic, Erin E. Krupa, Jeffrey C. Shih, Jonathan D. Bostic, Erin E. Krupa, Jeffrey C. Shih

• Publisher: Routledge
• Year: 2019
• Print ISBN: 9781138598713
• eBook ISBN: 9780429942112
• Edition: 1
• Topic: Education
• Subtopic: Education General

This book aims to provide theoretical discussions of assessment development and implementation in mathematics education contexts, as well as to offer readers discussions of assessment related to instruction and affective areas, such as attitudes and beliefs. By providing readers with theoretical implications of assessment creation and implementation, this volume demonstrates how validation studies have the potential to advance the field of mathematics education. Including chapters addressing a variety of established and budding areas within assessment and evaluation in mathematics education contexts, this book brings fundamental issues together with new areas of application.

1
Introduction

Aims and Scope for Assessments in Mathematics Education Contexts: Theoretical Frameworks and New Directions

Jonathan D. Bostic, Erin E. Krupa, and Jeffrey C. Shih

The aim of this edited book is twofold. First, it provides readers with a deeper understanding of validity, validity evidence and arguments, and the validation process; differing approaches to the validation process; and authentic examples of ways to convey validation arguments for assessments. The book also includes chapters addressing a variety of established and budding areas within assessment and evaluation in mathematics education contexts. Thus, its audience is intended to include anyone who conducts assessment and measurement work in mathematics education contexts.

The American Educational Research Association, American Psychological Association, and National Council on Measurement Education ([AERA, APA, & NCME], 2014) provide clear guidelines regarding measurement validity and reliability in the Standards for Educational and Psychological Measurement in Education (hereafter referred to as the Standards). Sufficient evidence for five sources must be shared related to validity: (1) content evidence, (2) evidence for relationship to other variables, (3) evidence from internal structure, (4) evidence from response processes, and (5) evidence from consequences of testing (AERA et al., 2014). Unfortunately, “evidence of instrument validity and reliability is woefully lacking” (Ziebarth, Fonger, & Kratky, 2014, p. 115) in the mathematics education literature. Worse yet, evidence related to the validity of quantitative assessments and measures has not necessarily been conceptualized or defined consistently in the research literature (Lissitz & Samuelsen, 2007; Mislevy, 2007). In the last 10 years, there has been a concerted response by some working within the mathematics education research space to address this omission; these authors present validity arguments and purpose statements and discussion of them in an effort to better support research with quantitative measures (see AERA et al., 2014; Kane, 2016, 2001; Pellegrino, DiBello, & Goldman, 2016; Schilling & Hill, 2007; Wilson, 2005).

Readership and Intentions

This book has the potential to be used in a variety of contexts and by a wide audience. It should be noted for readers that this book has a complement—Quantitative Measures in Mathematical Knowledge: Researching Instruments and Perspectives (Bostic, Krupa, & Shih, 2019)—which focuses on validity arguments related to mathematics content measures. We, the editors, purposefully wanted two separate books because assessments and measures are different in nature (AERA et al., 2014). A test or measure is “an evaluative device or procedure in which a systematic sample of a test taker’s behavior in a specified domain is obtained and scored using a standardized process” (AERA et al., 2014, p. 224), whereas an instrument may or may not involve correct or incorrect answers but rather Likert scales, indicators, or other criteria (Krupa, Carney, & Bostic, 2019). Assessment is broader than an instrument or measure (AERA et al., 2014). The Standards (AERA et al., 2014) define assessment as “any systematic method of obtaining information, used to draw inferences about characteristics of people, objects, or programs; a systematic process to measure or evaluate the characteristics or performance of individuals, programs, or other entities, for purposes of drawing inferences” (p. 216) or concomitantly as “a process that integrates test information with information from other sources (e.g., information from other tests, inventories, and interviews; or the individual’s social, education, employment, health, or psychosocial history)” (p. 2). Hence, assessment includes a broader and more inclusive notion than measure or test. To that end, this book focuses on assessments for use in mathematics education contexts.

Second, we, the editors, intend for this book to be widely accessible to readers, including academic professionals, graduate students, industry experts working within the educational space, as well as practitioners. We are particularly excited about two areas of potential readership. The first is that this book may serve those working within the mathematics education space as well as related fields such as learning sciences, cognitive science, psychometrics, research assessment and evaluation, policy, special education, and other fields. Given a wide readership, a goal from this book and its complement is to encourage synergistic work across diverse scholars, which results in knowledge that has strong intellectual merit and broader impact. As editors, we are also excited to support those who are new to assessment. Reviews of assessment work within mathematics education research (e.g., Beckman, Cook, & Mandrekar, 2005; Bostic, Krupa, Shih, & Carney, 2019; Bostic, Lesseig, Sherman, & Boston, in press; Boston, Bostic, Lesseig, & Sherman, 2015; Hill & Shih, 2009) indicate that validity rarely comes up in peer-reviewed mathematics education scholarship. Moreover, when it does, the validity evidence centers on content and internal structure evidence. Connecting validity to these two sources of evidence is woefully lacking in building a robust validity argument that supports valid interpretations and uses and does not adhere to current Standards (AERA et al., 2014), much less prior Standards (AERA, APA, & NCME, 1999). To that end, we intend for graduate students, postdoctoral scholars, practitioners, and seasoned assessment veterans to reflect on their current assessment development practices and how this is linked with modern Standards (AERA et al., 2014).

A chapter by Lavery, Jong, Krupa, and Bostic (2019, this volume) provides readers with a broad overview of the validation process and ways to gather evidence. Examples are shared to instantiate the subprocesses during validity evidence collection, specific to each validity source. With greater volume and quality of work being done in this area, we anticipate seeing more manuscripts being submitted to peer-reviewed journals. In turn, this may cause some journal editors to reevaluate what is and is not appropriate for potentially publishable manuscripts, specifically the value of validity arguments related to assessments used within mathematics education research and the interpretations drawn from those assessments.

Validity and Validation

It is expected that assessments, broadly speaking, develop a purpose statement for users, and convey a validity argument. A purpose statement provides important information about an assessment—and readers will find examples of them within this book. Validity is not a dichotomous notion; instead, it spans a continuum. Validity is “the degree to which evidence and theory support the interpretations of test scores for proposed uses of the tests” (AERA et al., 2014, p. 11). Put simply, how confident can assessment users feel that the interpretations from an assessment’s data can be trusted?

The Standards describe five sources of validity evidence: content, response processes, relations to other variables, internal structure, and consequences from testing. The amount of validity evidence needed to believe the outcomes and interpretations does not necessarily equate to evidence for all five sources, but it also does not equate to evidence from just one or two sources. For instance, decades of mathematics education research grounded validation arguments for assessments in test content (e.g., expert panel) evidence and/or internal structure (e.g., exploratory or confirmatory factor analysis). While a group of experts might agree that an assessment is connected to an intended construct, does it elicit the intended responses in appropriate ways (response process)? The factor structure and internal structure of an instrument might be satisfactory, yet the consequences from its use may have serious negative consequences that outweigh the benefits. Thus, assessment developers and users should consider some questions as they design and revise assessments:

1. What is the intended use for the assessment?
2. What evidence is needed to convey to others that the assessment does what it is designed to do?
3. How should that evidence be gathered?

In that sense, some readers may sense connections between validation and backwards design (Wiggins & McTighe, 2005). If an assessment developer knows how the assessment will be used, for what purposes, and what interpretations may be drawn from assessment data, then the design decisions during assessment development are appropriately guided by a roadmap of sorts (AERA et al., 2014; Newcomer, 2012; Wilson, 2005; Wilson & Wilmot, 2019 [this volume]). Answering these questions may lead to gathering evidence for some but not all of the sources. Additionally, it is likely to lead to data collection during multiple rounds. For instance, assessment developers might gather evidence for content, response process, and internal structure during one round of data collection. A second or several subsequent, follow-up study or studies may take place to gather evidence related to relations to other variables and test consequences. These evidence pieces come together to form a validity argument. Walkowiak, Adams, and Berry (this volume) describe two separate studies within their chapter to provide evidence alongside two validation frameworks: Schilling and Hill’s work (2007) and the Standards (AERA et al., 2014). Through multiple studies and a distinct focus on explicating the validity argument, Walkowiak and her colleagues provide one instantiation of how to combine frameworks to tell a comprehensive story about two assessments’ outcomes. Moreover, their work suggests how assessment developers are not confined to using one validity framework. Instead, assessment developers might creatively weave evidence together across two frameworks to better ground their validity arguments regarding assessments’ outcomes.

A validity argument is drawn together much like a mathematical proof, as a form of argumentative writing. These arguments are built on the complexity, quality, and frequency of validity evidence (AERA et al., 2014; Bostic, 2017; Kane, 2012, 2016; Jacobsen & Borowski, [this volume]; Lavery, Jong, Krupa, & Bostic, 2019; Walkowiak et al., 2019; Wilson & Wilmot, 2019 [this volume]). Wilson and Wilmot provide readers with a description of how the Standards (AERA et al., 2014) might be used in conjunction with the Bear Assessment System (Wilson, 2005) as a way to develop and frame validity arguments. There is neither one way to convey validity evidence to an intended audience nor one validation argument framework that is consistently better than others—it is up to the assessment developers to present their ideas in a coherent and consistent fashion (see Nilsson & Ryve, 2010 for a discussion of coherence and consistency). This volume contains several descriptions of how validity argument frameworks might be utilized to convey information about the results and conclusions from assessments.

We contrast argumentative writing with persuasive writing because one style of writing is more likely to be found within a validation argument. The former uses logical chains of ideas to communicate a rationale that claims and evidence are reasonable. Arguments may acknowledge limitations and share delimitations and are intended to convince a reader that ideas are sound. Persuasive writing, in contrast, intends to persuade a reader that one position is better than another in some regard. There is no other position to claim as being better when compared to an alternative; hence, persuasive writing is not purposeful for validation work. Argumentative writing aims to convince the reader using appropriate evidence and connections that an outcome was logically drawn. To that end, validity arguments offer readers a chance to thoughtfully consider an assessment developer’s idea, its limitations, and discern the degree to which it is coherent. No validity argument is perfect; however, there are stronger ones based upon the frequency and type of evidence, and the quality of its purpose statement, given for an assessment.

Validation, as a process, provides developers and potential users with information about the degree to which one can trust the outcomes and interpretations of an assessment. It is essential for anyone engaging in quantitative research to consider the validity evidence and overall validation argument for an assessment that might be used. Kane (2016) reminds readers that validation should not be left to academics only—as those working in schools (e.g., teachers, curriculum specialists, principals, and other staff) should consider whether the information derived from an assessment logically flows from it. “Validation may not be easy, but it is generally possible to do a reasonably good job of [it] with a manageable level of effort” (Kane, 2016, p. 79). We agree that validation should be part of any study using an assessment meant to generate quantitative data, which includes but is not limited to surveys, concept inventories, measures, and observation protocols.

Validity evidence for validity arguments is tied to the outcomes and interpretations of an assessment; an assessment is not valid (AERA et al., 2014). An assessment is validated for use within particular contexts (i.e., settings, durations, and populations) and use outside of those specified contexts puts the interpretations at risk of being misinterpreted and inappropriately drawn (AERA et al., 2014; Kane, 2012; Newcomer, 2012). Validation is not something completed once and never examined again. It is a process, and validity arguments should be reevaluated when warranted (e.g., new population of respondents, revisions or updates to an assessment, or change in the level of stakes of interpretations). In this sense, as Jacobsen and Borowski (this volume) communicate, validation has merit as a methodology within mathematics education work. Similarly, Bostic, Matney, Sondergeld, and Stone (2019) argue that validation is akin to design-based research (e.g., Middleton, Gorard, Taylor, & Bannan-Ritland, 2008) because of their similarities. Generally speaking, validity arguments include a series of logical claims and inferences about the assessment. There are numerous frameworks for grounding validation frameworks (e.g., Kane, 2012; Mislevy, Almond, & Lukas, 2003; Pellegrino et al., 2016; Schilling & Hill, 2007; Wilson, 2005). This book provides examples of how assessment developers might overlay the Standards with published frameworks (see Walkowiak et al., this volume; Wilson & Wilmot, this volume). Because the sources of validity evidence function to some degree as buckets to fill (Lavery, Holloway-Libell, Amrein-Beardsley, Pivovarova, & Hahs-Vaughn, 2016), the Standards’ expectation of gathering evidence to address sources of validity evidence meshes well with multiple validation argument frameworks. The key with any validity argument is that the necessary and sufficient evidence is provided. Weisstein (n.d.a) frames necessary as “a condition which must hold for a result to be true, but which does not guarantee it to be true.” Similarly, Weisstein (n.d.b) characterizes sufficient as “a condition which, if true, guarantees that a result is also true.” Thus, the decision about necessary and sufficient information to link claims and evidence within a validity argument for a particular assessment rests with the assessment developer’s intentions. Hence, assessment development is not something to take lightly, perform quickly, or without purpose.

A central outcome from this book and its companion is filling a needed gap within the scope of mathematics education research with a resource for end-users and assessment developers describing quantitative measures for mathematics education. Assessment developers may feel more confident in their actions during the design phrases. Assessment users may feel empowered during the assessment selection process to think through their needs and assessments’ intended uses—and to ask critical questions about the assessments they might use in practice and/or research. This book provides both theoretical narratives and practical examples about linking notions of validity with...

Table of contents

• Cover
• Half Title
• Series Page
• Title
• Copyright
• Contents
• List of Contributors
• Acknowledgments
• 1 Introduction: Aims and Scope for Assessments in Mathematics Education Contexts: Theoretical Frameworks and New Directions
• 2 Developing an Instrument With Validity in Mind
• 3 Measure Validation as a Research Methodology for Mathematics Education
• 4 Gathering Validity Evidence Using the BEAR Assessment System (BAS): A Mathematics Assessment Perspective
• 5 Validity Arguments for Instruments That Measure Mathematics Teaching Practices: Comparing the M-Scan and IPL-M
• 6 Design and Validation Arguments for the Student Survey of Motivational Attitudes towards Statistics (S-SOMAS) Instrument
• 7 Measuring Self-Efficacy to Teach Statistics in Grades 6–12 Mathematics Teachers
• 8 Measurement and Validity in the Context of Mathematics Coaches
• Index