Quantitative Measures of Mathematical Knowledge

Researching Instruments and Perspectives

Jonathan Bostic, Erin Krupa, Jeffrey Shih, Jonathan Bostic, Erin Krupa, Jeffrey Shih

• Publisher: Routledge
• Year: 2019
• eBook ISBN:
  9780429942235
• Topic: Education
• Subtopic: Education General

The aim of this book is to explore measures of mathematics knowledge, spanning K-16 grade levels. By focusing solely on mathematics content, such as knowledge of mathematical practices, knowledge of ratio and proportions, and knowledge of abstract algebra, this volume offers detailed discussions of specific instruments and tools meant for measuring student learning. Written for assessment scholars and students both in mathematics education and across educational contexts, this book presents innovative research and perspectives on quantitative measures, including their associated purpose statements and validity arguments.

1
Validation in Mathematics Education

An Introduction to Quantitative Measures of Mathematical Knowledge: Researching Instruments and Perspectives

Erin E. Krupa, Jonathan D. Bostic, and Jeffrey C. Shih

Quantitative tools generate data resulting in analyzable numerical values that allow for statistical (inferential) analysis (Measurement in Mathematics Education Working Group, 2016). It is critical for a quantitative tool to measure the construct it intends to measure. Connections between a latent construct and how it is operationalized must be strong; otherwise, research foundations have potential to rest on spurious notions or unnecessary variance. For example, recently research indicated that free and reduced-price lunch (FRPL), which is often a measure of socioeconomic disadvantage, is actually not a valid operationalization for socioeconomic disadvantage relating to household income but may predict some type of educational disadvantage (Domina et al., 2018). It is important that quantitative measures have undergone a rigorous validation process for a similar reason. Without clear evidence to support the intended uses of a measure, claims made based on the measure are worthless, unsubstantiated, and potentially misinforming research. In her handbook chapter on research in mathematics education, Confrey states, “cumulatively the field loses credibility when researchers fail to meet the standards for rigor in research” (Confrey, 2017, p. 15). We argue the standards for the rigorous validation of measures’ outcomes and careful attention to the uses of those measures in mathematics education need more attention by the research community, practitioner audiences, graduate education of future scholars in the field, and peer-reviewed journal articles.

The aim of this edited book and its companion, Assessments in Mathematics Education Contexts: Theoretical Frameworks and New Directions (Bostic, Krupa, & Shih, 2019), is to share descriptions of quantitative measures within mathematics education contexts, including purpose statements and validation arguments. A purpose statement is a concise narrative that describes the intent of the instrument and supports the use of the instrument in a particular context (Kane, 2001, 2012, 2016). In the Standards for Educational and Psychological Measurement in Education (Standards), the American Educational Research Association, American Psychological Association, and National Council on Measurement Education (2014) state, “A sound validity argument integrates various strands of evidence into a coherent account of the degree to which existing evidence and theory support the intended interpretation of test scores for specific uses” (p. 21). Chapters in this book weave together various validity evidence, from prior and current research efforts, to present sound validation arguments for interpretations of conclusions from specific quantitative measures.

A key focus in these chapters and the Standards is on interpretation of test scores for specific uses and not on the misinformed phrasing of “validity of the test”. It is quite common among educational researchers and practitioners to refer to the validity of a test (Bostic, Krupa, Carney, & Shih, this volume). Further, there are reports from the larger field of educational scholarship that current conceptions of validity and validation are not widely used (Cizek, Rosenberg, & Koons, 2008; Shear & Zumbo, 2014; Wolming & Wikström, 2010). This book includes chapters addressing a wide range of quantitative measures within assessment and evaluation in mathematics education contexts and draws upon current conceptions of validity and validation in mathematics education. A central goal of this volume is to fill a needed gap in the field with a resource describing quantitative measures for mathematics education and their associated validity arguments. Thus, its chapters may be examples for others to follow, across diverse disciplinary backgrounds and career stages, as well as valuable descriptions of measures with validation arguments.

Importance of Validity

Measure quality strongly influences the quality of data collected and relatedly, findings of a research study (Gall, Gall, & Borg, 2007). Measures with a clearly defined purpose and supporting validity evidence are foundational to conducting high-quality quantitative work (Newcomer, 2009). A review of mathematics education research using quantitative measures presents a noticeable challenge: there are few syntheses of quantitative assessments for mathematics educators to employ and even fewer discussions of the validity evidence necessary to support the use of assessments in a particular context (Bostic et al., this volume). Quantitative measures lacking validity and reliability evidence may generate spurious results (Gall et al., 2007; Wilhelm, Gillespie, & Jones, 2018). Moreover, using a measure lacking validity and reliability evidence across multiple studies may lead to a research foundation built upon spurious results due to a flawed measure. It is advantageous for the scholarly community of mathematics educators and those exploring mathematics education phenomena to seriously consider their instrumentation and data collection as being grounded in a robust validation argument.

The heart of any methodology is the measure or instrument used to collect data (Newcomer, 2009). Scholars conducting quantitative research typically choose to either use a preexisting measure or develop a new one, depending on the purpose of the research. Informed decision making comes from having available data about quantitative measures. The Standards (American Educational Research Association et al., 2014) provide detailed guidelines regarding measurement validity and reliability. At a minimum, sufficient evidence for five sources must be shared related to validity: evidence from test content, evidence from response processes, evidence from internal structure, evidence for relationship to other variables, and evidence from consequences of testing (American Educational Research Association et al., 2014; Gall et al., 2007)—or a strong rationale for why evidence for one of those variables is unnecessary. For example, imagine a high school student taking a 10-item, constructed-response geometry problem-solving measure. If the measure uses items that have little or no content validity, then there is only a slim chance that the measure provides useful data about the student’s geometry problem-solving performance. As another example, imagine that a mathematics content measure has low reliability; then the overall score may likely show greater evidence of random guessing than actual mathematics knowledge (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing (U.S.), 1999; American Educational Research Association et al., 2014; Hill & Shih, 2009; Shavelson, 1996). Additionally, if two students of equal ability but different ethnicities both take the same measure and score differently, then there is an issue of validity evidence related to other variables (in this case, ethnicity). Having any, or all, of these issues with validity evidence raises concerns about the interpretations of results from the measure. These examples also highlight the need to carefully and critically examine the validity and reliability evidence of measures used in research.

Unfortunately, “evidence of instrument validity and reliability is woefully lacking” (Ziebarth, Fonger, & Kratky, 2014, p. 115) in the literature. Validation studies of quantitative measures are noticeably absent from mathematics education journals too, which presents the challenge of determining whether an instrument is appropriate for a given study, much less whether its interpretations and results will be useful for analysis (Bostic et al., this volume; Hill & Shih, 2009). For instance, Hill and Shih (2009) reported that 8 of 47 studies published in the Journal for Research in Mathematics Education (JRME) in a 10-year time period provided any evidence related to validity and the majority provided only psychometric evidence. More recently, Bostic and colleagues (this volume) examined articles using quantitative research from two sources: JRME between 1970–2017 (n = 97) and a specific domain, namely early childhood research articles collected by the Development and Research in Early Math Education (DREME) Network between 2014–2018 (n = 24). They found most articles reporting student outcomes present no validity evidence (88%) related to the outcomes. Further, they report that of the 12% that did discuss validity evidence, the majority only mentioned test content and internal structure, and that when validity was discussed it was typically in reference to the measure, rather than the interpretation of the score.

Syntheses of measures for use in mathematics education can be found in the literature but are typically not intended as a comprehensive analysis. For example, Carney, Brendefur, Hughes, and Thiede (2015) conducted a brief review of self-report instructional practice survey scales applicable to mathematics education. The review was intended to provide a background on existing measures and their associated validity evidence in relation to a new measure under development. Boston, Bostic, Lesseig, and Sherman (2015) conducted a review of three widely known classroom observation protocols to assist mathematics educators in determining the appropriate tool for their particular research question and context. More recently, Bostic, Lesseig, Sherman, and Boston (in press) completed a synthesis of classroom observation protocols used in the last 25 years, which were appropriate for K-12 settings. It is important that this type of synthesis work continues and is encouraged by the field, which includes action by journal editors, university researchers, agencies that provide funding for research, and training for graduate students regarding how to conduct syntheses and metanalyses. In addition, we need to consider even more comprehensive approaches to ensure a breadth of topics and measures are examined.

In order for research in mathematics education to become more cumulative and to build on previous research, research instruments need to be widely shared, easily accessible, consistent in their intended use, and uniform in the interpretation of scoring. This book provides a much-needed resource for scholars making decisions about the use and interpretations of available quantitative measures, and it provides a comprehensive discussion of instruments used to measure students’ mathematics content knowledge (Arneson, Wihardini, & Wilson; Confrey & Toutkoushian; Kosko; Perry & Ketterlin Geller, & Hatfield; Melhuish & Hicks, all this volume), mathematics instruction (Carney, Totorica, Cavey, & Lowenthal, this volume), and mathematics teachers’ knowledge (Matney, Bostic, & Lavery, this volume).

Further, there are two chapters devoted to current trends in quantitative measurement of teachers’ and students’ competencies and future directions of that work (Bostic, Krupa, Carney, & Shih; Howell, Stone, & Kane, both this volume). Bostic and co-authors critically examine measures of students’ knowledge and then raise particular opportunities for scholars to pursue in future research. Howell and colleagues identify challenging aspects of creating a validity argument and describe new approaches to assessing measures of teachers’ competencies. They present three trends around teachers’ competencies and describe important implications of these trends in relation to the construction of validity arguments.

Validity Arguments and Validation Evidence

This book and its companion, Assessments in Mathematics Education Contexts: Theoretical Frameworks and New Directions (Bostic, Krupa, & Shih, 2019), are intended to be accessible to a wide readership. These books are part of the dissemination efforts from a National Science Foundation funded conference, Validity Evidence for Measurement in Mathematics Education (V-M² Ed) (NSF #1644314), which brought together over 40 educational researchers working in mathematics education research with expertise in mathematics education, psychometrics, applied measurement, and special education to construct a shared understanding regarding validity within mathematics education contexts. Conversations centered on the perspective of validity articulated in the Standards (American Educational Research Association et al., 2014), including a strong focus on the meaningful operationalization of constructs into quantitative variables, and on Kane’s (2006) model for articulating the purpose of an instrument with the proposed interpretation of test scores and necessary evidence to support the stated interpretations.

Five Sources From the Standards

Many of the chapters in this book link validity evidence to the five sources presented in the Standards: content, response processes, internal structure, relations to other variables, and consequences of testing (American Educational Research Association et al., 2014). Below we describe how authors in the chapters present evidence regarding the five sources. Note these are presented as examples to highlight some of the ways that this volume presents validity evidence connected with the five sources; however, the methods are neither exhaustive nor comprehensive. We encourage readers to examine each chapter for a more robust understanding of evidence regarding the five sources.

Test Content

Test content refers to “the relationship between the content of a test and the construct it is intended to measure” (American Educational Research Association et al., 2014, p. 14). Melhuish and Hicks (this volume) present a validity argument for a concept inventory for the Group Theory Concept Assessment (GTCA) that links validity evidence they collected to the five sources. In a very detailed and logical validity argument, they present five claims, one for each source of validity evidence, with subclaims for each that provides detailed explanation and results regarding the validity evidence they collected, and it supports their claims. One unique contribution of this paper is the use of a Delphi Study (Dalkey & Helmer, 1963) in the validity argument for evidence related to test content. Utilizing a Delphi Study, experts in group theory completed a series of rounds to determine the standard topics covered in an introductory group theory course. Unlike an expert panel, after each round, the experts received a summary of responses from the entire group and were given time to reflect and respond to the summary, which provided them with time to clarify their views and consider alternatives.

Response Processes

The second source, response processes, is evidence that connects how test takers may respond to a test item and how they actually respond to the item. In creating an assessment to measure teachers’ knowledge of the Standards for Mathematical Practice (CCSSI, 2010), Matney and colleagues (this volume) gathered evidence based on the Standards. They conducted cognitive interviews with a sample of teachers who might complete the instrument, namely pre-service and in-service teachers as a source of response processes evidence. The cognitive interviews provided them with rich data regarding how the interviewees interpreted the items and offered ideas for revisions to the instrument.

...

Table of contents

• Cover
• Half Title
• Series Page
• Title
• Copyright
• Contents
• List of Contributors
• Acknowledgments
• 1 Validation in Mathematics Education: An Introduction to Quantitative Measures of Mathematical Knowledge: Researching Instruments and Perspectives
• 2 The Form of Mathematics in Assessment Items: How Items Convey and Measure Multiplicative Reasoning Differently
• 3 Substantiating Claims About Students’ Algebraic Reasoning: Initial Evidence Based on Response Processes and Internal Structure
• 4 A Validation Approach to Middle-Grades Learning Trajectories Within a Digital Learning System Applied to the “Measuring Characteristics of Circles”
• 5 Assessing College-Ready Data-Based Reasoning
• 6 A Validity Argument for an Undergraduate Mathematics Concept Inventory
• 7 Developing a Construct Map for Teacher Attentiveness
• 8 A Validation Process for Complex Knowledge: The Standards for Mathematical Practice Knowledge Assessment
• 9 Reflecting on the Past and Looking Ahead at Opportunities in Quantitative Measurement of K-12 Students’ Content Knowledge
• 10 Future Directions in the Measurement of Mathematics Teachers’ Competencies
• Index