Designing Data Reports that Work

A Guide for Creating Data Systems in Schools and Districts

Jenny Grant Rankin

• Publisher: Routledge
• Year: 2016
• Edition: 1
• eBook ISBN:
  9781317353331
• Topic: Education
• Subtopic: Education General

Designing Data Reports that Work provides research-based best practices for constructing effective data systems in schools and for designing reports that are relevant, necessary, and easily understood. Clear and coherent data systems and data reports significantly improve educators' data use and save educators time and frustration. The strategies in this book will help those responsible for designing education data reports—including school leaders, administrators, and educational technology vendors—to create productive data reports individualized for each school or district. This book breaks down the key concepts in creating and implementing data systems, ensuring that you are a better partner with teachers and staff so they can work with and use data correctly and improve teaching and learning.

1 Data Reports Are the Silent Star

DOI: 10.4324/9781315665849-1

The Data Analysis Error Epidemic

The National Center for Education Statistics estimates less than 2 percent of school districts in the U.S. are able to turn data¹ into information educators can actually use (Sparks, 2014). Worse, educators who do use data typically use it incorrectly. In national studies of districts known for strong data use, teachers showed difficulty with question posing, data comprehension, and data interpretation, answering only 48 percent of questions correctly when drawing inferences from given data (U.S. Department of Education Office of Planning, Evaluation and Policy Development [USDEOPEPD], 2009, 2011).

That occurred at the nation's top data-using districts, whereas educators’ data use elsewhere is worse. For example, in a quantitative study involving 211 educators of varied backgrounds and roles, participants averaged 11 percent correct when answering basic analysis questions about given data (Rankin, 2013). This latter study took place in California, where multiple state assessments have a history of rendering data that is difficult to understand. Many studies rendered similar findings: educators use data to inform decisions, but they commonly misunderstand the data. Whether data-using conditions are ideal or not, an accuracy range of 11–48 percent when educators use data is alarming.

Here is a synopsis of how educators’ data-informed decision-making impacts students:

Figure 1.1

Educators Cannot Maximize Data without Data Reports that Work

Most literature and dialogue in education communities mistakenly treats data misunderstandings as if they are educators’ fault. If educators really were to blame, it would be within educators’ power to entirely remedy data problems. However, even when educators implement recommended interventions with fidelity, this is not possible.

For example, professional development (PD) and staffing-based supports such as strong leadership, data coaches, professional learning communities (PLCs), and collaboration hold great potential to improve educators’ data use, and most districts already utilize them on some level. However, many findings indicate PD and staff supports are not omnipotent. In other words, while they are beneficial and recommended, they will not bring a 100 percent accuracy rate to educators’ data analyses. For example:

• Many districts cannot give schools the amount of PD they need; PD might not always be available locally due to budget cuts, and developing PD resources on their own is too expensive an option for schools and districts (Kidron, 2012).
• In a study where teachers received PD in educational measurement/assessment, all teachers struggled afterwards with statistical terms and measurement concepts (Zapata-Rivera & VanWinkle, 2010).
• A report prepared for IES by Regional Educational Laboratory Midwest confirmed the limitations of the most popular data analysis supports in school districts, noting data staff and training resources can be limited at the local level, as is staff with proper data analysis experience and skills at the state level (McDonald et al., 2007).
• Teacher coaches can stop coaching teachers as the school year progresses due to other responsibilities (Underwood et al., 2008).

  “Minimize the extent to which [educators] need to actively analyze data. … We give such tools to physicians and military decision makers; education is no less complex and no less important.”

  U.S. Department of Education Office of Educational Technology, 2012, p. 49

• Expecting educators to be dependent on on-site data system experts for help to colleagues causes problems and creates a bottleneck and can leave support staff overworked (Wayman et al., 2007).

Even educators employing PD and staffing-based supports with fidelity cannot solve the data analysis error epidemic on their own. Educator-implemented interventions can improve educators’ data use, but they can only help to some degree.

Educators Are Primed for Good Data Use

Those providing data systems and/or data reports to educators have been given some favorable circumstances with which to work. Though no one is perfect, as is no teacher preparation program, consider how ideal educators generally are as a client- or user-base. Educators are primed for good data use in that they are:

• highly skilled—e.g., 95 percent of teachers are considered “highly qualified” by No Child Left Behind (NCLB) standards (American Institutes for Research, 2013);
• well educated—e.g., 99 percent of American teachers have bachelor's degrees, 48 percent have master's degrees, and over 7 percent have more advanced graduate degrees (Papay, Harvard Graduate School of Education, 2007);
• intelligent—e.g., educators have above-average IQs;²
• care about students—e.g., 85 percent of teachers say they became teachers because they wanted to make a difference in children's lives (Bill and Melinda Gates Foundation, 2014), and 90 percent of students believe their teachers care about their learning (Northwest Evaluation Association, 2014);
• embracing data use—e.g., most educators are eager to analyze and then act on the data they see (Hattie, 2010; van der Meij, 2008);
• embracing technology use—e.g., teachers indicated overwhelming support for using technology to improve learning, and 85 percent of teachers reported daily use of technology to support teaching (Bill and Melinda Gates Foundation, 2012);
• employing approaches within their power to grow professionally—e.g., districts devote 1 percent to 8 percent of their operating budgets to providing professional learning (Killion & Hirsh, 2012); the quality of training and supports can always be improved, but at least some is taking place.

With such highly qualified and proactive individuals making so many data analysis errors even at districts known for strong data PD, staff, and culture, educator-implemented data analysis interventions are clearly not enough. It is important to consider and remedy problems with the reporting tools educators are using.

Effective Data Reporting Is Often the Missing Ingredient

Providing a data system designed specifically for users’ needs is more effective than expecting training to get users as prepared as they need to be to use the system and its data (Underwood et al. 2008). Expecting educators to improve data use while using ineffective or marginally effective data tools is misguided.

In education, turning to tools for support with data analysis is less common than turning to leaders or PD (Marsh et al., 2006). Yet problems with current data systems and data reports undermine educators’ data use:

• In a study involving interviews with education leaders in 30 school districts and charter management organizations serving approximately 2,500 to 25,000 students each, administrators reported (a) challenges in getting data out of data systems undermine data-informed instruction, and (b) it is difficult to find useful data in their systems that can be analyzed in meaningful ways (Freeland & Hernandez, 2014).
• In the same study, some school systems reported being so dissatisfied with data systems and dashboards on the market that they built their own in-house data systems (Freeland & Hernandez, 2014).
• According to Thomas Kane, a professor at Harvard Graduate School of Education and a faculty director for the Center for Education Policy Research, data analysis problems are often not related to the data not being collected; rather, they relate to the data not being organized in a way that can answer questions, which are answerable but not being asked (Davis, 2013).
• When it comes to data systems at the state level, most attention goes toward technology aspects such as hardware and software, and any reports they generate for educators do not answer questions they could already answer without the system (Data Quality Campaign, 2011).
• Score reports designed specifically for administrators are frequently not designed in ways that are easy for administrators to interpret (VanWinkle et al., 2011).
• Teachers often underutilize data systems, finding available data to be unsatisfactory or late, having a hard time locating needed data, and finding the system to be hard to use (Cho & Wayman, 2012).
• 67 percent of 4,600 surveyed teachers indicated they were not satisfied with the digital data tools they use (Bill and Melinda Gates Foundation, 2015).
• A study by the U.S. Department of Education found data systems are frequently of limited use in informing instruction because of limitations in terms of the system's data, interface, or tools (Faria et al., 2012).

There are many types of data reporting tools. It can help to consider one as an example. One of the most common types of data system, in which some data reports are commonly embedded, is the student information system (SIS). In a survey and focus group study involving 716 district, school, and technology leaders and 1,010 teachers, Gartner and the Bill and Melinda Gates Foundation (2014) found:

• Dissatisfaction—Only 46 percent of teachers, 35 percent of school leaders, and 47 percent of district leaders are satisfied with their SIS.
• Limited Reporting—Only 51 percent of teachers believe the SIS helps them monitor student progress; only 19 percent of school leaders and 11 percent of district leaders believe it supports state reporting/policies; and only 4 percent of district leaders identify simplified reporting as one of their SIS benefits.
• Inefficient—Only 37 percent of teachers believe their SIS aids school efficiency, and only 8 percent of district leaders identify increased efficiency as one of their SIS benefits.
• Cumbersome—Only 29 percent of teachers believe their SIS lets them spend more time on teaching and less on administrative tasks, and only 29 percent of tea...

Table of contents

• Cover
• Half Title Page
• Title Page
• Copyright Page
• Table of Contents
• Preface
• Acknowledgments
• Meet the Author
• eResources
• 1 Data Reports Are the Silent Star
• 2 Label Standards
• 3 Supplemental Documentation Standards
• 4 Help System Standards
• 5 Package/Display Standards: Big Picture
• 6 Package/Display Standards: Report Design
• 7 Package/Display Standards: User Interface
• 8 Content Standards
• 9 Work with Educators
• 10 Put It All Together