How to Make Decisions with Different Kinds of Student Assessment Data

Susan M. Brookhart

• Publisher: ASCD
• Year: 2015
• ISBN:
  9781416621065
• Topic: Education
• Subtopic: Professional Development

In How to Make Decisions with Different Kinds of Student Assessment Data, best-selling author Susan M. Brookhart helps teachers and administrators understand the critical elements and nuances of assessment data and how that information can best be used to inform improvement efforts in the school or district. Readers will learn—
* What different kinds of data can—and cannot—tell us about student learning;
* What different analyses reveal about changes in student achievement;
* How to interpret, use, and share relevant data; and
* How to create a model to go from problem to solution in a data-based decision-making process. With easy-to-understand explanations, supplemented by examples and scenarios from actual schools, this book offers a path to better understanding, more accurate interpretation of assessment results, and—most important—more effective use of data to improve teaching and learning.

---

Chapter 1

---

An Introduction to Different Kinds of Data

. . . . . . . . . . . . . . . . . . . .

---

---

Simple logic does not always help us with data interpretation. Here are two examples, both of which are true stories.

Once during a workshop session, a school administrator explained to me that he saw standardized test scores as a sort of barometer. He aimed to make changes in his school that would lead to a rise in test scores, and that would be his indication that his reforms were successful. The analogy he gave me was the population of oysters in the Chesapeake Bay. Environmental reforms were needed in the bay area, changes were made, and the oyster numbers are increasing (see http://www.chesapeakebay.net/issues/issue/oysters for more information). Similarly, he explained, reforms in his school should lead to higher test scores.

The second example occurred at an airport, where I struck up a conversation with a businessman and his young son who were waiting for the same airport shuttle I was. The man said that he was really glad that his state now administered standardized tests, as per No Child Left Behind, because finally he had what he called a "bottom line" that he could watch to know how his child and his child's school were doing. His analogy was to the bottom line in a profit-and-loss statement in a business.

The administrator and the businessman dad were well-meaning people who valued education; they were not nay-sayers. They were both bright and successful individuals who applied logic and common sense to a problem they cared about. And they were both wrong.

Here's the thing. If you are an environmentalist or an oysterman, the oysters are the issue—or in the case of the Chesapeake Bay, one of the issues. Increasing the oyster population, to improve the habitat and the economy, is the purpose of the scientific reforms and management strategies. More oysters means the program is achieving its goal. Similarly, generating a profit is the purpose of being in business. Higher profits mean more money for shareholders, employees, and product development. Making money means your business is achieving its goal.

In contrast, raising test scores is not the purpose of education. The purpose of education has changed with society's needs and values over the years (Sloan, 2012). At this point in time, if you ask people the purpose of education, you will get answers such as these: to create adults who can compete in a global economy, to create informed citizens who can participate in the democratic process, to create critical thinkers and problem solvers, to create lifelong learners, or to create emotionally healthy adults who can engage in meaningful relationships. Obviously, no test score can tell you whether you have achieved these things.

The less obvious problem with our well-meaning administrator and businessman is that even if you limit your interest to academic learning outcomes, raising test scores is not the purpose of education. The students' learning is. Test scores are a measure of student learning, but they are not the thing itself. In our analogies, the oysters and the money were themselves the objects of interest. You can count oysters, and you can count money, but you can't "count" learning.

The best you can do to measure learning is to use a mental measurement that, if well designed, is a measure of learning in a limited domain. The key is to define clearly what that domain is, use a test or performance assessment that taps this domain in known ways, use a score scale with known properties that maps the student's performance back to the domain, and interpret that score scale correctly when making inferences about student learning. The purpose of this book is to explain just enough about the properties of data on student learning so that you can make those inferences well. Then—and only then—can you make sound decisions. Another name for this purpose is developing assessment literacy. As the examples demonstrate, literacy in educational assessment involves more than counting or ranking. It involves specifying what specific learning you are measuring; understanding how the questions or tasks in the measure form a sample of that domain of learning; understanding properties of the scales, numbers, or categories used in the measurement; and being able to reason from all these things to make sound interpretations and decisions.

To complicate matters a bit, as the title of this book indicates, there are different kinds of data. For most educational decisions, you will want to mix the different kinds of data to broaden and deepen the pool of information about student learning that you use to monitor and improve that learning. You will want to know which kinds of data to watch, and when, in order to evaluate the effectiveness of your decisions. This book will help you do that in two ways. First, it offers a framework for thinking about assessment systems that categorizes different measures of student learning. Understanding how information differs from one category to another will help you interpret data. Second, this book offers some insights into different types of scores. Understanding different types of scores and their meanings will help you interpret data properly, as well. Equipped with an understanding of these two big ideas, your data interpretation and subsequent decisions will be more sound, more valid, and more useful.

The Purposes and Uses of Data

The phrase "data-based decision making" is used often and has many meanings. Teachers use data to answer questions about students. Groups of teachers and building administrators use data to answer questions about students, classes, programs, and their school. Central office administrators use data to make decisions about teachers, as well as students, classes, programs, and schools. An Internet search on "data-based decision making" will bring up dozens of PowerPoint presentations, PDFs, images, and plans. Many books also address this theme.

It may seem like an obvious point, but the data you choose should be related to the intended purpose. If I want to make a tablecloth, I need to measure the length and width of the table; measuring its height won't help me much. The same principle operates in making decisions about student learning, but it's less easy to see. For example, if I want to make a decision about which reading skills to emphasize in my reading class, shouldn't I just look up students' scores on the state reading test? No. What the state reading test measures is general, overall "reading achievement," as defined by a whole set of reading standards. State test results will give you a sense of how your students do at "reading in general," as defined by whatever reading standards your state says its test measures, taken all together.

For example, the Smarter Balanced Assessment Consortium says that, regarding reading, its assessments can support this claim: "Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts" (Smarter Balanced, 2014). If I'm a teacher with a student whose assessment results suggest he can't do that very well, how do I design instruction for him? Low test results relative to this claim suggest a general decision—more or different reading instruction—but don't provide any clues about what aspects of reading to emphasize, remediate, or build on. To design reading instruction for that student, I'll need different data, assessment data that give a more fine-grained description of what the student can and cannot do as he reads.

In this simple example, reasoning from data is a two-step process. Data from the standardized accountability assessment help me identify a problem (Arland doesn't read proficiently) and lead me to another question: Why? To answer that second question, I need different data, because the reading accountability test doesn't give me information that is specific enough. Using complementary kinds of data for educational problem solving requires understanding different kinds of data.

This focus on a deeper understanding of data about student learning is what sets this book apart from other data books. I will, of course, also talk about how to use the data to inform instructional improvement. Two other excellent books that talk about using data to inform instructional improvement are Data Wise by Boudett, City, and Murnane (2013) and Using Data to Focus Instructional Improvement by James-Ward, Fisher, Frey, and Lapp (2013).

This book complements those and other books about data by focusing on developing a clearer understanding of exactly what test scores and other data about student learning are and what they mean. As an analogy, think about reading a Shakespeare play in a high school English class. If you read the play with a basic understanding of the English language, you will understand the plot. If you take the time to learn some Shakespearean vocabulary, you will understand the plot and the word-play nuances that unlock some of the humor and characterization in the play. In other words, you will understand the play better. Similarly, if you do data-based decision making with a basic understanding of assessment and of numbers, you will be able to make general decisions. If you learn some concepts about how educational assessments are constructed and some nuances about what their results mean, you will understand better what the data are telling you about your students' achievement.

Data About Student Learning

One way to organize and describe the different kinds of data about student learning is to use a four-quadrant framework (Brookhart, 2013). This framework allows us to group different kinds of data according to general type and purpose and to examine how they complement each other. It gives us some vocabulary to describe "assessment" in more specific terms. Figure 1.1 shows a four-quadrant framework for describing different kinds of assessments of student learning. The framework does not attempt to cover other data of interest to educators (e.g., student attendance, the number of books in the library, the ratio of students to teachers), just assessments of student learning.

---

FIGURE 1.1 Four-Quadrant Framework for Describing Types of Assessments

---

This framework will help you use different types of data to get richer, fuller information for your classroom decisions. The response of teachers and administrators to this framework has been quite positive. I have found that people are looking for a way to describe their "assessment system" that is more than just a long list of assessments.

Two dimensions: Purpose and focus

The framework in Figure 1.1 defines two dimensions on which assessment of student learning can be described: intended purpose for the information (formative or summative) and intended focus of the information (classroom or large scale). Of course, individual students are the ones who are assessed in all cases; even the large-scale state accountability test is administered to individual students. The focus dimension is about the place where the information is centered, and for large-scale assessments that focus is across individuals, classrooms, and schools.

Purpose. If learning is the main emphasis in education, then the distinction in purpose—between assessment that informs learning and assessment that certifies that learning—is important. Many readers will be familiar with balancing formative and summative assessment in their classroom practice. Formative assessment, or assessment for learning, occurs during learning and is intended to result in improved learning (Moss & Brookhart, 2009). Summative assessment, or assessment of learning, occurs after an episode of learning and is intended to summarize the student's achievement level at a particular time (Moss, 2013). Typically, formative assessment items and tasks, and formative feedback, tackle next-step-sized learning targets. By the time summative assessment is appropriate, the outcome may be broader. As a somewhat oversimplified illustration, feedback on a 2nd grader's writing might be about capitalization and punctuation today and ideas tomorrow, and the final graded writing sample may appraise both.

Focus. The location of reference for the learning information is the other dimension—whether assessment is centered in the classroom or in a large-scale context. Some readers may be less familiar with distinguishing classroom-focused from large-scale assessments than they are with distinguishing formative and summative assessment purposes. After all, it's students in classrooms who take all the assessments, right? In Chapters 2 through 5, you will see that it is very useful to distinguish assessments that are primarily focused on the learning that occurs in one classroom, with its particular instructional context, from assessments that are primarily focused on generalizing across classroom contexts. The two differ in important ways, most notably on what specific learning is assessed and in the kinds of numbers that are used to quantify student performance. Some of that assessment information is meant for classroom use, and some is meant to be aggregated across classrooms for larger-scale evaluation—of a course, a program, or a school, for example.

Four quadrants

Crossing the two dimensions results in four quadrants that define the four major types of assessment of student learning that are used in schools, or what I have been calling "different kinds of data." These types are formative classroom assessment; interim/benchmark assessment, including "common formative assessments" that are intended to be given in more than one classroom; summative (graded) classroom assessment; and summative (accountability) large-scale assessment. I'll briefly describe each type here and then devote a chapter to each.

Formative assessment: Formative purpose, classroom focus. Formative assessment is an active and intentional learning process that partners teachers and students to continuously and systematically gather evidence of learning with the express goal of improving student achievement (definition from Moss & Brookhart, 2009, p. 6; also see Wiliam, 2010). Formative assessment involves strategies such as the following (Moss & Brookhart, 2009; Wiliam, 2010):

• Sharing learning targets and criteria for success with students
• Feedback that feeds forward, from teachers, peers, or other sources
• Student self-assessment and goal setting
• Using strategic questions and engaging students in asking effective questions

One of the hallmarks of formative classroom assessment is student involvement. Formative assessment strategies aim to develop assessment-capable students who can see where they are headed (they can envision a learning target and know what it represents), take stock of where they are in relation to the target, and understand what they need to do next to continue to approach the learning target. Formative assessment's foundation is the students' clear concept of a learning target, or even a broader learning goal, and a clear understanding of what achievement of that goal looks like. This means students understand the criteria for success, or what Moss and I call "student look-fors" (Moss & Brookhart, 2009). Receiving feedback and using it to improve, setting goals and monitoring progress toward them, and asking effective questions all are based on the foundation of understanding "what I am trying to learn."

In recent years, people have quibbled over whether students have to be involved in making decisions about assessment results in order for the assessment to be formative. One of the reasons for the confusion is ignoring the distinction between classroom-focused and large-scale assessment. Students have to be involved in classroom formative assessment, which works only when students take action on what they should do next in their learning. They can't take effective cognitive action if they aren't involved in making the decisions. However, students don't necessarily have to be involved in decisions made about large-scale assessments that are intended to be formative in purpose. Teachers may use these results to modify instruction, for example, without the students knowing about it.

I have found that the focus dimension—classroom versus large scale—helps enormously with the vocabulary problem educators have been struggling with regarding formative assessment. For example, I was talking with a principal who thought "formative assessment" had to refer to the interim assessments his district purchased from a testing company, and so he didn't know what to make of the formative assessment strategies that occur within daily lessons—which was the topic I was at his school to address. I showed him the four-quadrant framework, and he found it immediately helpful. We might not be able to do much about the fact that the term formative assessment is currently used in too many different ways, but we certainly can make sure that we understand exactly what we are talking about for any specific data and interpret the data accordingly. This framework will help you do that.

Interim assessment: Formative purpose, large-scale focus. Teachers can use interim/benchmark assessments that do not involve students—other than to respond to assessment items or tasks—to inform instructional planning for those students or even for future students. This is a formative purpose, although it's not what I generally have in mind when I use the term formative assessment; to me, that term usually means classroom formative assessment. Interim assessment and classroom formative assessment are diff...