Handbook of Data-Based Decision Making in Education

Theodore Kowalski, Thomas J. Lasley

• Publisher: Routledge
• Year: 2010
• Print ISBN:
  9780415965040
• eBook ISBN:
  9781135890834
• Topic: Education
• Subtopic: Education General

Education has fought long and hard to gain acceptance as a profession and, since professionals by definition use data to shape the decisions they make, education has little choice but to continue moving in this direction. This 3-part handbook represents a major contribution to the literature of education. It is a unique compendium of the most original work currently available on how, when and why evidence should be used to ground practice. It is a comprehensive, cross-disciplinary, research-based, and practice-based resource that all educators can turn to as a guide to data-based decision making.

The Handbook of Data-Based Decision Making in Education is a must read for researchers who are just beginning to explore the scientifically based nature of educational practice. It is also appropriate for policy makers and practitioners who are confronted with young people who need to be in classrooms where "best practices" are the norm and not the exception.

Part I: Theoretical and Practical Perspectives

1
Evidence and Decision Making in Professions

Theodore J. Kowalski
University of Dayton

Professional practitioners are expected to rely on scientific evidence to make crucial decisions affecting their clients or patients. This seemingly innocuous benchmark, however, has generated considerable controversy, especially regarding the extent to which intuition, emotion, politics, and philosophy should influence these choices. Traditionally, science (or more precisely, scientific research) has served the purpose of protecting “practitioners from implementing useless programs” (Mayer, 2003, p. 361). Physicians, for example, have been and still are required to pass comprehensive examinations on the medical profession’s scientific knowledge base as a licensing prerequisite. Today, however, the accumulation of massive amounts of data and the development of technology that allows databases to be accessed easily and quickly have spawned an accountability movement that is sweeping across professions. The overall goal of this movement is to make evidence-based practice (EBP) normative (Levant, 2005).

The application of EBP in education has not been debated widely and the extent to which educators understand and support the concept is unknown. Even so, administrators and teachers historically have exhibited a proclivity to oppose ideas that conflict with their prevailing beliefs, especially when these ideas emerge in the context of politically coercive change strategies (Bauman, 1996). As an example, critics of the No Child Left Behind Act of 2001 (NCLB)¹ have often reacted negatively to the law’s requirements for data-based decision making. More specifically, they have argued that basing consequential decisions solely on research data is demeaning and precarious—demeaning because the wisdom of educators is devalued and precarious because research data are fallible (Kowalski, Lasley, & Mahoney, 2008).

Given the role of schools in society, the future of EBP in education should not be determined solely by political, emotional, or even philosophical discourse. Rather, making the concept normative for administrators and teachers should depend on the extent to which it improves practice and ultimately, school effectiveness. The purpose here is to provide a framework for examining EBP in the context of professional responsibility. First, the topics of problem solving and decision making in schools are addressed. The intent is to (a) clarify the nature of these processes, (b) delineate the nexus between them, and (c) contrast programmed and un-programmed decisions. Next, evidence is defined and multiple types of evidence are identified and incorrect and correct interpretations of EBP, generally and in education, are discussed. Last, the barriers preventing EBP from becoming normative are categorized.

Professional Problems and Decisions

Though there are countless definitions of a profession, they are essentially “occupations with special power and prestige. Society grants these rewards because professions have special competence and esoteric bodies of knowledge linked to central needs and values of the social system” (Larson, 1977, p. x). Recognized professions enjoy a symbiotic relationship with societies; that is, in return for services rendered, their practitioners are granted influence and social status (Kowalski, 2004). Curtis (2000), citing the work of British ethicist, Paul Rowbottom, identified five characteristics of a profession:

• a theoretical knowledge base,
• protocols of practice,
• knowledge is developed via research and transmitted via publications by and among members,
• members accept and are held accountable to a service ethic, and
• academic preparation and entry are rigorous and controlled.

The scope of professional knowledge required for practice and the manner in which knowledge and skills are acquired have become increasingly important because society has come to expect that practitioners are near perfect in exercising authority. That is, society has become more intolerant of practitioners who err (May, 2001).

If practice in professions merely involved the application of scientific knowledge, pre-service technical training would be sufficient. Schön (1983, 1990) explains, however, that practitioners frequently encounter problems that defy textbook solutions. Studying the treatment of difficult problems, he concluded that highly effective practitioners possessed both theoretical and craft knowledge, the latter being a form of artistry acquired through practice-based experiences. As craft knowledge accumulated, practitioners were increasingly able to resolve or at least manage atypical problems. Scholars generally agree that intuition in the form of craft knowledge plays an important role in practitioner competency, especially with respect to producing and validating evidence. The discovery of penicillin and nylon demonstrates the validity of this conclusion. Both provide “well documented cases of the ‘intelligent noticing’ of evidence that emerged outside the intellectual infrastructure from which evidence is expected to materialize” (Thomas, 2004, p. 3).

In essence, craft knowledge provides a practitioner with the ability to think beyond the parameters of technical knowledge.

Problem Solving

In professions, practitioners encounter both routine tasks and problems. The former are situations that can be managed successfully by following prescribed or customary actions; they typically require some technical knowledge but little analysis. The latter are perplexing and unique situations characterized by intricate unsettled questions; they too require technical knowledge but also extensive analysis. Fundamentally, problem solving is an analytical process that entails making and evaluating decisions in relation to problems.

Experiences across professions have demonstrated that practitioners err when they take problems for granted (Nutt, 1989). Unproductive responses are most often rooted in incorrect or superficial perceptions of a situation. According to Heifetz (2006), misdiagnosing problems is one of the five most common mistakes made by administrators and it entails the failure to distinguish between technical and adaptive problems. He explained that the former “have known solutions and can be handled with authoritative expertise. But there are no readily available solutions for adaptive challenges—those that require new ways of thinking and behaving” (p. 512).

When a problem is defined incorrectly or incompletely, subsequent choices (decisions) made in relation to it are likely to be ineffective or possibly counterproductive. Consider a first-grade teacher who described a male student’s poor academic performance as “the expected outcome of a limited intellect.” Her view of the problem was affected by knowledge of the student’s family (e.g., his siblings were not good students and neither of his parents had graduated from high school) and his behavior in the classroom (e.g., he had a low attention span and was socially immature for his age). Her definition of the problem prompted her to make a series of counterproductive decisions. For instance, she decided to ignore or explain away evidence indicating that the student might be dyslexic; she stubbornly refused to refer the student to the school psychologist so that diagnostic tests could be administered; she set low expectations for the student and essentially treated his poor academic performance as being normal.

Because persons have a proclivity to define problems instinctively, cognitive psychologists recommend that three aspects of a situation should guide the framing process: (a) a current state, (b) a desired state, and (c) a lack of a direct obvious way to eliminate the gap between the current state and desired state (Mayer, 1983). In simple terms, a problem exists when something is needed or wanted but the decision maker is unsure what to do in order to attain it (Reys, Lindquist, Lambdin, Smith, & Suydam, 2003). A future state (or desired outcome) should be described using measurable criteria, ensuring that objective and accurate assessment is possible (Clemen, 1996).

After the problem is framed, a practitioner is expected to determine its level of difficulty. One approach for determining difficulty, developed by Reitman (1965), is a four-tier typology. The categories group problems based on current and desired states.

• Category 1 problems include situations in which the current and desired states are well-defined.
• Category 2 problems include situations in which the current state is well-defined but the desired state is poorly defined.
• Category 3 problems include situations in which the current state is poorly defined but the desired state is well-defined.
• Category 4 problems include situations in which both the current and desired states are poorly defined.

Routine problems have low difficulty; therefore, they can usually be addressed successfully by applying technical knowledge and authority. Conversely, difficult problems usually defy textbook solutions. Instead, they require adaptive solutions, actions that lead educators to challenge current practices and to experiment with non-traditional approaches (Heifetz & Linsky, 2002).

Decisions

Decision making, the core process in problem-solving, is basically a three-stage procedure:

1. Identifying choices (alternative decisions), demands (e.g., expectations, job requirements), and constraints (e.g., laws, policy, lack of resources).
2. Evaluating choices in relation to demands and constraints.
3. Selecting the best alternative.

Objective decisions depend on the decision maker’s ability to identify and apply criteria that define an acceptable decision (Kowalski et al., 2008). Without criteria, a person cannot rationally determine the merits of each alternative that is being contemplated, and thus, an emotional or political decision becomes more probable.

Ideally, practitioners want to minimize demands (e.g., pressure for a principal to make a decision favorable to one group) and constraints (e.g., eliminating some possible choices because they require additional resources) so that they can identify, assess, and evaluate as many choices as possible (Sergiovanni, 2006). When discretion is eliminated (i.e., a person has only one choice), a practitioner is relegated to functioning as a manager; that is, he or she merely needs to determine how a law, policy, or rule will be enforced. As examples, mandatory sentencing laws restrict judges and zero-tolerance policies restrict school administrators. Conversely, discretion (having multiple choices) allows administrators to exert leadership by focusing on what should be done to manage a problem (Yukl, 2006).

Decisions, like problems, differ in difficulty. According to Simon (1960), difficulty is determined by three variables and the basic questions they generate.

1. Frequency—How often is a decision made? Frequency ranges from “routine” to “unique.” Unique decisions are more difficult than routine decisions.
2. Configuration—To what extent is a problem clear and easily framed? Configuration ranges from “unstructured” to “structured.” Unstructured decisions are more difficult than structured decisions.
3. Significance—What are the potential consequences? Significance ranges from “unimportant” to “important.” Important decisions are more difficult than unimportant decisions.

Schön (1983, 1990) concluded that decision difficulty was affected by both problems and context. He described three intermediate zones of practice that explain dissimilar levels of difficulty.

1. Uncertainty—Problems encountered by practitioners frequently do not occur as well-informed structures. When uncertainty is high, the probability of decision alternatives succeeding or failing is typically unknown; hence, uncertainty increases decision difficulty.
2. Uniqueness—Problems encountered by practitioners are frequently unfamiliar in that they were not addressed in textbooks, do not comply with the standards of espoused theories, and do not recur with regularity in practice. When uniqueness is high, the decision maker typically has limited confidence in theoretical knowledge or routine practice; hence, uniqueness increases decision difficulty.
3. Value conflict—Problems encountered by practitioners are frequently characterized by competing values and beliefs. When value conflict is high, the decision maker knows that his or her choice is likely to be criticized by some individuals and groups; hence, value conflict increases decision difficulty.

Decision Analysis

Practitioners are expected to engage in decision analysis, a process that “provides structure and guidance for thinking systematically about hard decisions” (Clemen, 1996, p. 2). The intent of this scrutiny is to produce prescriptive advice that is especially helpful when a practitioner’s overall knowledge is insufficient to make an informed choice intuitively. According to Simon (1960), decision analysis requires an understanding of three different types of decisions.

1. Programmed—These decisions are routine, structured, and relatively unimportant; they can usually be made effectively by following pre-established policy and protocols.
2. Semi-programmed—These decisions are semi-routine, semi-structured, and moderately important; they can usually be aided by pre-determined policy and protocols.
3. Un-programmed—These decisions are uncommon, unstructured, and relatively important; they cannot usually be made effectively by following pre-determined policy or protocols.

A person’s understanding of decision difficulty is more complete when he or she evaluates decision choices in relation to contextual variables. Both Table 1.1 and Figure 1.1 summarize decision difficulty based on Simon’s (1960) decision characteristics and Schön’s (1983, 1990) intermediate zones of practice. Though all the factors are relevant to understanding decision difficulty, uncertainty arguably is the most critical variable because it directly relates to risk—that is, uncertainty increases the likelihood that an alternative will produce negative outcomes (Nutt, 1989).

Models are crucial to decision analysis because they provide both a framework and knowledge that is especially important to making un-programmed decisions (Kowalski, 2008). The literature on decision science identifies three categories of models. Dillon (2006) defined them in the following manner.

Table 1.1 Categorization of decisions based on difficulty.

Figure 1.1 Factors determining decision difficulty.

1. Normative models—These paradigms stipulate in theory what the decision maker should do.
2. Prescriptive models—These paradigms stipulate what the decision maker should and can do.
3. Descriptive models—These paradigms describe what decision makers actually have done.

There are a myriad of theories and models across these categories. Identifying and discussing all of them is not practical here; instead, the purpose is to demonstrate their value to practitioners.

Normative Models Traditionally, decision-making research has been grounded in two assumptions: it is an orderly rational process of choosing from alternative means of accomplishing objectives; it is a logical and sequential process (Owens, 2001). The classical paradigm, the quintessential example of a normative model, is a linear, scientific approach intended to produce the perfect or ideal decision. The classical model was routinely taught to professionals because it was considered to provide:

• rules for a potentially disorderly process,
• a deductive approach to problem solving, and
• predictability, order, technical competence, impersonality, and objective reasoning.

(Tanner & Williams, 1981)

The model was initially designed and deployed in economics in a context in which the decision maker acted as a “maximizer” (Razik & Swanson, 2002); that is, he or she quantified alternatives to determine the absolute best one.

Simon (1997), however, pointed out that the classical approach is based on a number of faulty assumptions. Most notably, it assumes that

• all possible decision alternatives can be identified (when in fact only a few can typically be identified),
• all consequences that would follow each alternative are known or could be accurately predicted (when in fact such knowledge is only fragmentary), and
• quantitative values can be assigned objectively and accurately to each alternative (when in fact values attached to consequences can only be imperfectly anticipated).

Other critics, for example, March (1978), add that decision makers are either unaware of their preferences or they avoid, suppress, or change them when selecting a preferred alternative. Moreover, ideal decisions are uncommon because decision makers typically are unable to (a) frame the problem perfectly, (b) ...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Preface
• Introduction: Contextualizing Evidence-Based Decision Making
• Part I: Theoretical and Practical Perspectives
• Part II: Building Support for Data-Based Decisions
• Part III: Data-Based Applications
• About the Authors