Toward Defining and Improving Quality in Adult Basic Education

Issues and Challenges

Alisa Belzer

• Publisher: Routledge
• Year: 2013
• Print ISBN: 9780415878067
• eBook ISBN: 9781135601317
• Edition: 1
• Topic: Education
• Subtopic: Adult Education

This volume revisits, problematizes, and expands the meaning of quality in the context of adult basic education. Covering a wide range of relevant topics, it includes contributors from the realms of both policy and practice and encompasses both the major instructional areas-reading, writing, and mathematics-as well as larger issues of literacy, learning, and adulthood. Each chapter focuses on what improving quality in the field might look like through the particular lens of the author's work. As a whole, the broad scope of topics and ideas addressed will raise the level of discussion, knowledge, and practice regarding quality in adult basic education. In this book, the term adult basic education refers to the broad range of services for adults who wish to improve their literacy and language skills, including beginning and intermediate writing, writing and numeracy, preGED, GED/Adult Secondary Education, and ESL instruction that takes place in a range of contexts including schools, community-based programs, and workplace development programs. The volume is organized around three themes:
*Accountability, Standards, and the Use of Documentation and Research;
*Program Structures and Instruction; and
*Rethinking Our Assumptions and Concepts.Coming at a time of increasing pressure to standardize, to be accountable, and to improve outcomes, and when calls for evidence-based practice are fueling stakeholders' interest in the relationship between research and practice at all levels of the system, Toward Defining and Improving Quality in Adult Basic Education is particularly timely for scholars, graduate students, and professionals in the field of adult basic education.

PART ONE

ACCOUNTABILITY, STANDARDS, AND THE USE OF DOCUMENTATION AND RESEARCH

Chapter 1

Accountability and Program Quality: The Third Wave

Larry Condelli

American Institutes for Research

Over the last 20 years, accountability has grown to be a central force in public policy. Driven by political concerns, uncertain funding, and pressure to resolve difficult social problems, the demand for publicly funded programs to demonstrate that their services lead to positive outcomes for their clients has increased steadily. At the federal level, demands for accountability led in 1993 to the passage of the Government Performance and Results Act, the first accountability legislation that required federal agencies to develop a set of performance measures to judge whether they were meeting their legislative mandates. Many states passed similar legislation for their education, welfare, and job training programs, often mandating that programs demonstrate successful client and programmatic outcomes, or risk financial sanctions or complete loss of funding.

In no area have pressures for accountability been greater than in education. A series of sharply critical studies of the state of education and failing schools led to the rise of a host of school reform efforts at the state and federal levels, which often included mandatory student testing and sanctioning of low–performing schools. Accountability demands for public schools culminated in 2002 with the enactment of the No Child Left Behind legislation, which included mandatory testing of all students.

Although lacking the politically charged atmosphere surrounding public schools, the adult basic education (ABE) system has faced similar demands for accountability. Mirroring the broader public policy arena, concerns over ABE program quality and demands for accountability arose in the late 1980s. In 1998, with the passage of the Workforce Investment Act (WIA), a state-level accountability system for ABE, with an incentive system tied to performance measures, was federally mandated for the first time. The National Reporting System (NRS), established by the U.S. Department of Education to implement the WIA accountability requirements, went into effect in July 2000.

Permeating debates around the value of accountability systems is the assumption–usually implicit--that they provide information about, and help improve, the quality of programs. For example, a school with high test scores is presumed better than a school with poor scores; an ABE program where most students pass the General Educational Development (GED) test is viewed as being of higher quality than one in which fewer students pass. Whether this assumption is true depends on the nature of the accountability system and how it is implemented.

This chapter explores the relationship between program quality and accountability in the context of federal efforts toward promoting accountability in the ABE program. These efforts have occurred in three distinct phases or waves over the last decade. In Wave 1, accountability focused on program inputs and process, whereas in Wave 2, student outcomes are the main focus. It appears that Wave 3, which is now emerging, will have elements of both processes and outcomes, balancing the two prior waves. The chapter discusses the approach toward accountability for each wave, evaluates its affect on program quality, and concludes with observations on how the accountability process can be improved to strengthen the accountability-quality link.

A FRAMEWORK FOR EVALUATING ACCOUNTABILITY AND PROGRAM QUALITY

To help us explore the relationship between accountability and program quality, I first describe general characteristics of accountability systems and review the literature on aspects of accountability systems that are most likely to affect program quality. I use this research as a framework to help us evaluate how federal approaches toward accountability in ABE have affected program quality.

Key Characteristics of Accountability Systems

As described by Stecher (1995) and Elmore (1997), a system of program accountability provides a systematic way to gauge whether a program is meeting its goals and mandates. Accountability systems can document how a program operates, or its processes; what a program produces or its clients achieve, or its outcomes; or both processes and outcomes. Most accountability systems currently in use are “top-down”–systems that have been legislatively mandated to serve the interests and policies of funding agencies. In these types of systems, policymakers and program directors set measures, methods, and procedures. Accountability systems also can be field driven, or “bottom-up.” Practitioners or service providers take a major role in devising the accountability system, under this approach.

Information from the accountability system is often used as a factor in funding decisions, although the data can also be used for program improvement efforts. Legislatively mandated, top-down approaches to accountability are usually more focused on the use of quantitative accountability data to inform a general system of performance-based funding. Bottom-up approaches typically have a more explicit and direct focus on program improvement, often include qualitative measures, with less emphasis on performance-based funding. A distinctive feature of the current trend in accountability has been the almost exclusive focus on outcome-based, top-down approaches, using quantitative measures.

The Job Training Partnership Act (JTPA) established the first major accountability system of this type. The JTPA program, which funded state and local adult job-training efforts from 1983 to 1998, required states and local programs to report on a range of participants’ employment outcomes. In their study of this accountability system, Baj, Sheets, and Trott (1995) used business management theory to identify four essential characteristics of all accountability systems:

• An underlying set of goals that the program is to achieve, such as promoting learning, literacy, self sufficiency or employment.

• A common set of measures that reflect the goals and that are reported and aggregated. The measures must be clearly defined and can be qualitative or quantitative.

• Performance standards tied to measures, which set a level or target of performance that programs must achieve.

• Sanctions or rewards for programs, tied to performance.

Only accountability systems that have all of these characteristics in place can successfully meet their goals of providing accurate measures of program performance and can positively influence program quality. To be implemented successfully, accountability systems need “buy-in” and acceptance from local programs, and sufficient resources to provide local programs with technical assistance and tools (e.g., computer hardware and software for data collection, documentation, and analysis) to make it feasible to implement requirements. Gaining acceptance is particularly important with “top-down” accountability systems that are defined and established by funding or oversight agencies for their grantees.

Accountability systems not only serve as a way to measure program performance, but they can affect program processes and outcomes. This dynamic nature of accountability systems makes them important policy tools: they can be used to implement policy and measure whether policies are effective. For example, emphasizing specific measures can define what stakeholders believe is important and focus programmatic efforts on improving performance on these measures. The same measures then become the means by which program success is evaluated. The clearest example of this dynamic is the emphasis on standardized testing in elementary and secondary education. Producing high test scores is a major focus of most schools, and high scores are the main criteria by which schools are judged (Elmore, 1997).

Accountability Factors Affecting Program Quality

Implicit in this approach toward accountability is that good performance means high program quality: a good program is one that has good performance measures. Most policymakers and the general public tend to believe in this relationship. However, outcome-based accountability systems usually lack direct measures of program processes or operations that define program quality. Little attention is usually given to how programs achieve these outcomes. Quality is only indirectly affected in this type of accountability system. Whether good outcomes, as measured through an accountability system, reflect program quality depends on how the four characteristics of an accountability system are implemented.

Over the last 15 years, several studies have explored how this policy-accountability dynamic affects local service quality by examining federal employment training programs, notably programs funded through the JTPA, the first major federal program to have an outcome-based accountability system (Condelli & Kutner, 1992; National Commission on Employment Policy [NCEP], 1987; Trott & Baj, 1996). These studies concluded that each of the four defining characteristics of accountability systems affect how local program services are provided and the success of the system in achieving the policy goals for the program.

The extent to which the accountability system reflects the goals of programs is central to how well the system can affect program quality. Although authorizing legislation defines programmatic goals for most social and education programs, a successful accountability system requires agreement between policymakers and practitioners on these goals and their priority in service delivery. For example, the level of importance of employment as a goal for ABE is often debated among policymakers, teachers, and employers. Due to its central importance, creating consensus on program goals among stakeholders is usually the first step in developing an accountability system.

Once the program goals the system will reflect are defined, one of the most difficult and contentious issues in the development of an accountability system is defining measures that match the goals. Although it may be relatively easy to build consensus that “literacy development” is a central goal of ABE, for example, determining how to measure it poses a much greater problem. Not only do measures faithfully reflect goals in an effective accountability system, but good measures also are closely tied to the services a program provides. A close link between services and outcome measures means that programs can more directly affect the measure and that the accountability system more accurately reflects program performance and program quality.

Performance standards in accountability systems are numeric targets that define the performance levels a program must meet to be judged a success. The level at which standards are set is a critical factor in whether the standards reflect program quality. Standards that are set too low are usually unable to affect quality because programs do not have to change what they do to meet them, and therefore do not need to take low standards very seriously. Standards that are set too high also do not usually improve program quality because they are too difficult to meet. Although there is often a tendency to set performance standards to high numeric levels in the belief that higher program quality will result, experience indicates that programs achieve excessively high standards by making changes that actually hurt program quality. For example, programs may enroll students with higher literacy levels who will progress faster and turn away lower literacy students; or programs could provide less comprehensive instruction that focuses on achieving quick outcomes. These “unintended consequences” can completely subvert the accountability system by preventing programs and students from achieving their goals and can end up actually reducing program quality. Finding the appropriate level at which to set standards to avoid unintended consequences usually means adapting them to local circumstances, such as the type of student enrolled and community conditions (NCEP, 1987).

Monetary incentives or sanctions are tied to performance standards to reward programs that exceed standards or punish those that fail to meet them. The amount of the rewards and sanctions can have a significant affect on program quality. In its study of the JTPA system, NCEP (1987) concluded that rewards and sanctions that are set too low are unlikely to affect quality, because they do not motivate providers to make programmatic changes. However, when they are set too high, rewards--and particularly sanctions--can greatly exacerbate unintended consequences. Faced with a significant or total loss of funding, for example, a program may resort to any number of strategies to achieve its numeric performance standards–strategies that can hurt or have little to do with program quality. Dishonest reporting is also more likely, as the recent case in the Houston public school system illustrates. Several schools exaggerated, and in at least one case falsified, data on student dropout rates due to fear about the sanctions involved in being labeled “low performing” (Winerip, 2003).

EVALUATING QUALITY THROUGH ACCOUNTABILITY IN ADULT BASIC EDUCATION

Research has demonstrated that outcome-based accountability systems can affect program quality through the program goals they define, the measures and performance standards they require, and the rewards and sanctions they provide to local programs (e.g., Elmore, 1997). Systems where stakeholders agree on goals that accurately articulate program purposes, that have measures that reflect these goals, and that are closely tied to the services programs provide, have realistic performance standards, do not have excessive sanctions or rewards for performance, and are most likely to affect program quality positively.

Using this research as a guide, I evaluate how federal approaches toward accountability in ABE have affected program quality. I first examine the U.S. Department of Education's initial attempt at accountability, which was not a formal system. It relied on a set of suggested, but not mandated, indicators to define program quality (Wave 1). My main focus is on Wave 2, the NRS which is the ABE program's formal accountability system, and on an upcoming third wave of accountability, legislative proposals to change the NRS. In my evaluation of the first two waves, I look at the purpose and rationale behind the accountability requirements, judge the success of their implementation, and reflect on their impact on program quality in ABE. In my discussion of the third wave, I speculate about the likely impact proposed changes will have on quality and suggest ways to enhance accountability to create a more explicit link between accountability and quality.

Wave 1: Quality Indicators

The ABE program's first legislative step toward formal accountability requirements appeared in the 1988 reauthorization of the Adult Education Act, which required states to evaluate local programs in six areas and to use standardized tests as part of their evaluation system. However, it was not until the passage of the National Literacy Act of 1991 that the Department of Education (ED) began its first major effort toward instituting an accountability system. The Act required states to develop “indicators of program quality” and to use them “to judge the success of [local] programs” as a way to assess program quality. The indicators were meant to measure “efficient and effective performance” of ABE programs in seven areas related to student outcomes and program processes: educational gains, recruitment, retention, support services, staff development, curriculum and instruction, and program planning. ED developed eight model indicators (two for educational gains) and required all states to use them, or to develop and use their own indicators, to evaluate local program effectiveness (U.S. Department of Education, 1992).

ED required states to define each indicator with a descriptive statement reflecting quality, and to attach measures to them. The model indicators and measures for curriculum and instruction and for retention, for example, included the following:

• Indicator: Program has curriculum and instruction geared to individual student learning styles and levels of student needs.

– Measures: Instructional content addresses individual student needs (measured by classroom observation or self-report); student goal setting processes are linked to instructional decisions.

• Indicator: Learners remain in the program long enough to meet their educational needs.

– Measures:Hours in program by learning gain; percentage of students returning after specified time period.

ED also encouraged states to develop performance standards for each measure. States were to use the measures as a means to assess program effectiveness, identify program improvement and technical assistance needs, and make funding decisions.

A review of states’ efforts in implementing quality indicators found that about two thirds of states developed and were using them in the intended ways. States reported that the development process itself had a positive effect on programs, as the process raised awareness of program quality issues and gave state and local staff the opportunity to define and reach consensus on the characteristics of effective, quality programs. Defining the measures and standards along with the indicators gave states direction on how to evaluate and improve local programs (Condelli, 1996).

Assessment of the Quality Indicators Approach

The quality indicator approach was not a fully realized system of accountability...

Table of contents

• Cover Page
• Half Title Page
• Series Page
• Title Page
• Copyright Page
• Contents
• Preface
• Introduction: Why Quality? Why Now?
• Part 1: Accountability, Standards, and the 9 Use of Documentation and Research
• Part 2: Program Structures and Instruction
• Part 3: Rethinking our Assumptions and Concepts
• Author Index
• Subject Index