Designing Performance Assessments for School Leader Readiness

Lessons from PAL and Beyond

Margaret Terry Orr, Liz Hollingworth

• Publisher: Routledge
• Year: 2020
• Print ISBN:
  9780367363048
• eBook ISBN:
  9781000090536
• Topic: Education
• Subtopic: Education General

In face of increased scrutiny on the preparation of educational leaders, this book provides a much-needed resource, exploring the role and use of authentic performance assessment for evaluating leader readiness and performance. Framed by theory and research on school leader performance assessment, Designing Performance Assessments for School Leader Readiness provides an in-depth description of one fully tested performance assessment called the Performance Assessment for School Leaders (PAL). The authors explore how to assess four components of leadership proficiency -- developing a plan for an area of school improvement, creating a professional learning culture among school staff, supporting individual teacher development, and engaging families and community in improving student learning. This book provides real examples and practical guidance on designing and managing performance assessment for aspiring educational leaders, and how the PAL can be used in regional, state and local contexts.

1
Overview of Authentic Performance Assessment

A new superintendent from a mid-sized semi-urban school district recently contacted the state education department’s division of licensure: “I am in the process of hiring several new principals to help improve our lower performing schools but am having difficulty evaluating applicants. What assumptions can I make about their leadership readiness based on their graduate preparation and your state’s licensure requirements? What evidence does the state use to determine readiness?” she asked. The state official paused, then said: “Well, we rely on the preparation programs to determine candidate readiness for licensure. We do not require a test or other evidence.” The superintendent responded with surprise: “You mean I need to find out which programs provide better preparation and infer that their graduates are more likely to be ready? I have to rely on a program’s reputation? No other indicator of readiness exists besides the letters of recommendation I receive from applicants?” She continued: “In my former state, we knew our applicants had passed a state test that measured basic leadership knowledge. Do you at least require that?”

This chapter presents an introduction to the concept of authentic performance assessments. It discusses the challenges of creating authentic tasks that yield valid results and the elements of a performance assessment. In addition, the chapter outlines the way that performance assessments can be used to evaluate leadership skills, particularly with respect to what researchers know about experiential learning theory.

The Challenge to Create Authentic Performance Assessments

Since 1996, the National Policy Board for Educational Administration (NPBEA)¹ established, and now recently revised, quality educational leadership standards to underscore expectations for the readiness and performance of school and district leaders that are central for school improvement and student learning (National Policy Board for Educational Administration, NPBEA, 2015). Yet these standards are insufficient without a means to assess how well current and aspiring leaders can meet them. Further, assessments must be sufficiently flexible to demonstrate candidate capacities in a variety of settings, because candidates must be ready to serve schools in many contexts and to serve students with diverse needs. In short, districts and states have a strong need for quality measures of initial leader readiness. They must be authentic, standards-aligned, skills based and comparable across a wide range of school, district and state contexts.

School district officials need independent evidence that aspiring candidates are ready for initial school leader positions. Without such evidence, they must rely upon minimum qualifications—a certificate or master’s degree in educational leadership and years of teaching experience—as well as letters of recommendation. While useful, this evidence is limited in its measurement of leadership skills around the essential work related to improving schools and is difficult to compare among candidates.

Using program completion as a proxy of candidate readiness is insufficient. Available evidence on leadership preparation programs suggests that they vary widely in their scope, content and rigor, despite state accreditation expectations (Black, Bathon, & Pointdexter, 2007; Darling-Hammond, Meyerson, La Pointe, & Orr, 2010; Orr, 2011). As a result of limited candidate readiness information, a preparation program’s reputation, rather than its graduates’ degree completion, becomes a proxy measure (albeit weak) of preparation quality. Most districts lack sufficient resources to develop their own means of assessing candidate readiness for a first-time leadership position. They, thus, must rely on state readiness measures and informal evidence, such as letters of recommendation, resumes, preparation program reputation and applicant interviews. Creating high-quality, psychometrically valid and reliable assessments can be a complicated, time-consuming undertaking.

Yet greater attention to creating standards-aligned, job-relevant and reliable assessments will strengthen both their quality and the field’s confidence in their use, making investment in their development worthwhile.

Through laws and regulations, states can establish qualifications for school and district leadership. All 50 states have adopted school leadership standards to guide related policies, but just 37 states require candidates to have earned a master’s degree and have three or more years of teaching (or related pupil personnel, such as guidance counselors) experience to earn initial school leader certification (Scott, 2018). Thirty-six states require candidates to pass a leadership exam, such as the School Leader Licensure Assessment (www.ets.org/sls/) or a state-developed test or assessment. Only Massachusetts and California require a performance-based assessment, rather than one that is cognitively based, which means their aspiring candidates must demonstrate their leadership skills to qualify for licensure and first-time leader positions.

Finally, aspiring leadership candidates need comparative information to benchmark their proficiency and readiness for initial school leader positions, beyond the feedback they might gain from internship supervisors and faculty as part of their leadership preparation program. Educational leadership standards set direction for aspiring leaders, but only aligned assessments (and their rating rubrics) can signal performance expectations that candidates could use to self-assess readiness or seek comparative feedback.

The Promise of Performance Assessments

Performance assessments are a promising approach to determining leadership quality and readiness in public education because they meet the need for information on candidate proficiency in relevant areas. Standards-based performance assessments that have been validated as job-related and context-relevant, and shown to be reliable, can signal expectations for aspiring leaders and yield comparable information for school districts and states in determining qualifications and school leader readiness.

Performance assessments have long been used in K-12 education to facilitate student learning and in evaluating candidate and staff proficiency and readiness in technical professional fields such as medicine and health care. Only recently have they been adopted for use in the educating professions, primarily in a range of teacher certification areas (Lalley, 2017). Given the complexity of school leadership and the limitations of more conventional, cognitive-based (knowledge-based) assessments of professional readiness (Grissom, Mitani, & Blissett, 2017), there has been a significant call for robust forms of assessment to determine leadership quality and proficiency (Vogel & Weiler, 2014).

Graduate-level leadership preparation programs seeking national accreditation recognition are required to use evidence from performance-based assessments. Few valid and reliable tools and resources exist, however, leaving programs to rely on locally constructed assessments for national accreditation reporting. Typically, they have poor validity and reliability, do not yield results that can be compared across programs and may be insufficient indicators of leadership skills and readiness.

Consequently, educational leaders and policymakers would benefit from guidance, such as this book, on the design and use of performance assessments in school leadership and the theory and research that undergird the potential use and effectiveness of performance assessments when well constructed.

The Concept and Elements of Performance Assessments

There are several definitions and uses of performance assessments in education. While widely used to measure learning in children, assessments have only recently become commonly used in education and licensure in some professional fields. For this book, we are using Wiggins’ (1993) definition of performance assessment. He explains succinctly that it consists of two parts. First, candidates engage in “worthy problems or questions of importance” (p. 228). Second, candidates demonstrate proficiency by using “knowledge to fashion performances effectively and creatively” (p. 228).

McTighe (2015) expands this definition by outlining seven characteristics of tasks to be completed for complex, authentic performance:

• Tasks require application of knowledge and skills, not just recall or recognition of information, yielding a tangible product or performance as evidence.
• Tasks are open-ended and can be accomplished in different ways; there is no one right answer.
• Tasks are performed under unique conditions and constraints, within authentic contexts.
• Tasks require higher-order thinking and the thoughtful application of knowledge and skills in context.
• Tasks are complex, requiring multiple steps and skills to complete.
• Tasks are evaluated with “clearly defined and aligned criteria [that] enable defensive, judgment-based evaluation” and rubrics that detail different proficiency levels.

Given these qualities, performance assessments are educative experiences as well, requiring candidates to engage in novel problem-solving as they figure out how to apply new knowledge and skills to complete the tasks. As Messick (1994) explains further: “Exposure to authentic assessment is expected to provide the student with a meaningful educational experience that facilitates learning and skill development as well as deeper understanding of the requirements and standards for good performance” (p. 17).

Performance assessments represent the form of assessment that measurement specialists recommend (Linn, Baker, & Dunbar, 1991). Linn et al. assert that performance assessments “seek to attend to complex learning and processes” and thus must have higher fidelity to its goals and standards, than indirect measures (p. 16). This fidelity includes both validity and reliability, as well as technical assessment adequacy. The authors propose the following criteria as a framework for the validity and use of performance assessments:

• Consequential considerations. They include the evaluation of the intended and unintended consequences of the assessments, such as priority setting for preparation and time use.
• Fairness of the uses and interpretations made of the test results. Assessments take account of diversity of backgrounds, experiences and cultural contexts.
• Transfer and generalizability. Evaluation of these characteristics can be improved by increasing performance assessment task elements and task specification according to critical dimensions.
• Cognitive complexity. Performance assessments shift the emphasis from factual knowledge application to the processes that candidates undertake for problem-solving and reasoning.

In addition to an assessment’s need to meet the criteria for validity, its content must be relevant to the elements of a leadership role. According to Linn et al. (1991), “The content needs to be consistent with the current understanding of the field and at the same time reflective of what are judged to be aspects of quality that will stand the test of time” (p. 19). This characteristic can be determined by involving subject-matter experts in the design review. Such a review should consider the following:

• Does the assessment content cover a broad representation of the elements of a leadership role?
• Are the tasks meaningful, in the sense that are they relevant and “worthy of attention” (p. 20)?
• Is the assessment cost-effective and efficient?

The Unique Qualities of Performance Assessment

Performance assessment has emerged as the optimal means of evaluating candidates’ knowledge, skills and readiness with respect to a leadership position because of its applied nature and capacity to evaluate multiple skills in use. It can be used for formative or summative purposes in preparation programs, or as a state school leader licensure assessment. Most important, performance assessment has benefits that render it superior to other types of assessments, both for the school or district see...

Table of contents

• Cover
• Half Title
• Title
• Copyright
• Contents
• Preface
• Acknowledgments
• 1 Overview of Authentic Performance Assessment
• 2 Leadership Needs, Preparation and Assessment: Policy and Practice
• 3 Practices for Assessing the Readiness and Proficiency of Education Leaders
• 4 The Massachusetts Performance Assessment for Leaders: Background and Components
• 5 Performance Assessment Scoring and Evaluation
• 6 The Design Considerations of a Leadership Performance Assessment
• 7 Measurement Considerations for Performance Assessment
• 8 The Benefits of Performance Assessment for Leadership Candidates, Preparation Programs, Schools and Educational Institutions
• 9 A Multiplicity of Uses for Leadership Performance Assessment
• 10 The Transformative Nature of Authentic Assessment for School Leaders
• Appendix A: PAL Rubric Alignment to the Massachusetts Department of Elementary and Secondary Education Professional Standards for Administrative Leadership
• Appendix B: Alignment of Educational Leadership Constituent Council Requirements and Massachusetts PAL Tasks
• Appendix C: PAL Task Instructions
• Appendix D: Rubric Example
• Appendix E: Massachusetts PAL Validity and Reliability Results
• Appendix F: Examples of Annotated Scoring With Evidence
• Index