Design-based Research

Clarifying the Terms. A Special Issue of the Journal of the Learning Sciences

Sasha A. Barab, Kurt Squire

• Publisher: Psychology Press
• Year: 2016
• Topic: Psychology
• eBook ISBN:
  9781135066284
• Subtopic: Cognitive Psychology & Cognition
• Index: Psychology

Over a decade ago the concept of "design experiments" was introduced because of the belief that many of questions could not be adequately addressed by laboratory-based experiments. Since then, design-based research as a term has grown in popularity and significance. The core manuscripts of this special issue respond to the questions: What constitutes design-based research? Why is it important? What are the methods to carry it out? At the end of this issue, two strong commentaries situate this work and challenge the community with new questions and issues that must be answered if design-based research is going to help advance work in ways that others judge as worthwhile and significant.

Creating a Framework for Research on Systemic Technology Innovations

Barry Fishman, Ronald W. Marx, Phyllis Blumenfeld, and Joseph Krajcik

School of Education
The University of Michigan

Elliot Soloway

College of Engineering
The University of Michigan

Correspondence and requests for reprints should be sent to Barry Fishman, 610 East University, Room 1360E, Ann Arbor, MI 48109–1259. E-mail: [email protected]

This article examines why cognitively oriented technology innovations, designed to foster deep thinking and learning, have not become widespread in K–12 schools. We argue a key reason is that most design-based research does not explicitly address systemic issues of usability, scalability and sustainability. This limitation must be overcome if research is to create usable knowledge that addresses the challenges confronting technology innovations when implemented in real-world school contexts. This is especially important in an era when political forces push schools away from the cognitively rich, inquiry-oriented approaches espoused by the Learning Sciences. We suggest expanding our conception of design-based research to include research on innovations in the context of systemic reform as a potential solution to the problem. To that end, we introduce research questions and issues arising from our own experiences with a technology-rich innovation in the context of a systemic reform initiative as a starting point in the creation of an expanded design-based research agenda. These questions and issues have important implications for both the continued viability of research on technologies for learning and on the future of technology use in schools that stems from such research.

Over the past decade, the Learning Sciences have built on knowledge of how people learn (Bransford, Brown, & Cocking, 1999) and made major investments in the design and development of learning environments that employ technology to foster thinking and understanding with demonstrated positive effects on learning (Roschelle, Pea, Hoadley, Gordin, & Means, 2000). Despite the fact that technology is now considered commonplace in K–12 education (Becker, 1999), most innovations derived from Learning Sciences research, which we refer to as cognitively oriented technology innovations, have not found their way into widespread classroom use. Instead, for a variety of reasons including teacher capabilities (CEO Forum on Education and Technology, 1999), technology infrastructure (Carvin, 2000), school culture (Cuban, 1986) and organizational constraints (Cohen, 1988), the primary uses of technology in schools remain drill and practice, word processing, and web surfing (R. E. Anderson & Ronnkvist, 1999). These uses of technology may be important initial steps for schools, but they fall short of the tremendous potential of technology to support the rich, inquiry-oriented learning called for in national standards documents (e.g., American Association for the Advancement of Science, 1993), and embodied in Learning Sciences research (Bransford et al., 1999).

Appropriately, the public demands a great deal from its education investment. Technology is an expensive and therefore popular target for criticism, particularly because demonstrating widespread benefits from its use in schools is a challenging problem for research (Means, Wagner, Haertel, & Javitz, 2000). Researchers have long understood that for technology to contribute to learning requires much more than simply installing it in schools. As one research group puts it, in order to be effective, “technology needs to be part of a coordinated approach to improving curriculum, pedagogy, assessment, teacher development, and other aspects of school structure” (Roschelle et al., 2000, p. 78). In other words, the most effective uses of technology are interwoven with the challenges and problems of school reform itself. However, much of the design-based research that is targeted toward the development of cognitively oriented technological innovations has focused on classroom-level or multi-classroom testbed implementations (Gomez, Fishman, & Pea, 1998) and not the larger contexts in which innovations are, or are not, actualized when used across entire school systems.

This article explores the question of why there is not more widespread use of technologies to foster deep learning aligned with national standards, and how we, as a research community, can learn more about how to foster, sustain, and scale these uses of technology. In particular, we explore the use of cognitively oriented technologies within systemic reform contexts, using our own experiences to frame new research questions and issues for exploration that shed light on the problems of creating scaleable, sustainable, and usable technology innovations. Additionally, we shed light on interrelating system variables that should be considered when conducting design-based research.

Why are cognitively oriented technology innovations not widely used in schools? Why aren’t they scaleable or sustainable? We believe an underlying explanation to be that we, as a scholarly community, have not focused our research on the development and use of cognitively oriented technologies in a way that ad dresses the fundamental needs of school systems. Instead, research on cognitively oriented learning technologies has focused primarily on students, teachers, and classrooms as the primary unit(s) of analysis. Though we recognize the need to link technology and reform, the field lacks a bridge between focused research and development of learning technologies and the broad-based systemic use of these innovations in schools. Shepard (2000) recognized this as problem for the broader educational research community in her AERA Presidential Address, when she advised researchers to develop methodologies that embrace “dilemmas of practice.” Such work “would advance fundamental understandings at the same time that they would work to solve practical problems in real-world settings” (p. 13). This focus would lead to the production of more readily “usable knowledge” (Lagemann, 2002). As researchers, we have developed rich understandings of how technology can foster learning in specialized situations; we now need to develop knowledge about widespread appropriation and use of cognitively oriented technologies by schools and school systems as part of real-world reform efforts. Ultimately, this calls for an augmented research agenda designed to enhance the usability of technology innovations developed by the research community, with positive consequences for scalability and sustainability. To address this issue requires that we introduce new questions and ways of thinking about problems into our research agendas. The framework that guides these questions must combine the best of what we currently understand about learning and teaching with technology with what is already known about the challenges of creating systemic reform and the implementation of innovations in reform contexts.

We begin by clarifying the nature of cognitively oriented technology innovations in school and research contexts. Next we reflect on what is known about fostering the widespread use of technology innovations in schools, and provide a framework for considering the usability of innovations. We then turn to research questions arising from our own experiences in working in systemic reform to frame a systemic research agenda that addresses issues of sustainability, scalability, and usability of cognitively oriented technology innovations, concluding with reflection on issues for the research community that are related to carrying out our proposed research agenda.

COGNITIVELY ORIENTED TECHNOLOGY INNOVATIONS

In our thinking, cognitively oriented technology innovations focus on inquiry and approaches to learning as embodied in national standards documents (e.g., National Council of Teachers of Mathematics, 1989; National Research Council, 1996). These innovations include technology as a core component, but are rooted in cognitive and constructivist learning theories (Bransford et al., 1999; J. S. Brown, Collins, & Duguid, 1989). Cognitively oriented technology innovations range from intelligent tutoring systems that help students learn mathematics (e.g., J. R. Anderson, Corbett, Koedinger, & Pelletier, 1995) to environments that foster communal knowledge-building and support for writing (e.g., Scardamalia & Bereiter, 1991) to tools that scaffold deep explorations in science (e.g., Linn & Hsi, 2000). In these innovations, technology is employed as a tool to support teaching and learning, as opposed to the object of learning. These innovations often use technology to scaffold teaching and learning practices that would be difficult to achieve otherwise, such as making complex causal modeling accessible to students (e.g., Jackson, Stratford, Krajcik, & Soloway, 1994).

There is a continuum of ways that technology is employed in cognitively oriented technology innovations. Instruction can be delivered via computer, as is the case with intelligent tutoring systems, or computers can be used as resources and “learning partners” in classrooms where much learning takes place “off line.” Cognitively oriented technology innovations may be designed to cover a relatively short period of time, or they may be comprehensive, intended to be used throughout an entire year or across multiple years of instruction. Our focus, however, is on cognitively oriented technology innovations that are closely tied to the regular curriculum and tightly integrated with teaching and learning practices. In instances where this is not the case, such as after-school computer clubs (e.g., Zhao, Mishra, & Girod, 2000), there may be high-quality learning and excellent uses of technology, but we do not include these instances in our consideration of cognitively oriented technology innovations for teaching and learning because such extracurricular uses of technology side-step the challenges of systemic reform and are not designed or intended to influence teaching and learning by teachers in regular school subjects as part of the school day.

The Knowledge Integration Environment (KIE; Bell, Davis, & Linn, 1995; Linn & Hsi, 2000) is an example of a cognitively oriented technology innovation. KIE combined a range of networked software tools with constructivist pedagogical principles in order to foster use of evidence and argument in middle school science. In KIE, learning is organized around generative questions in science, such as whether light travels forever or dies out. Students conduct research collaboratively on the Internet to gather evidence or view evidence developed expressly for KIE, and use argument-support software to organize their evidence and make supportable claims. Students then debate their claims and use of evidence in face-to-face classroom discussion as well as through the use of asynchronous on-line discussion tools. The technological aspects of the learning environment in KIE are designed to complement the face-to-face learning facilitated by the classroom teacher, who must orchestrate collaboration, guide students in their learning, and provide assessment and feedback to students. KIE is challenging for teachers in that they need to understand not only the content embedded in this project (which is just one of several in a broader KIE-enabled science curriculum), but also how to help students use the technology, how to foster collaboration, and how to conduct appropriate assessments. As with many cognitively oriented technology innovations, access to the Internet is crucial. If the Internet connections or computers are not working, it is difficult to use KIE curriculum. In this way, cognitively oriented technology innovations are often demanding of both the instructional and technology infrastructures of schools. Recently, research on KIE has evolved into a larger effort called the Web-based Inquiry Science Environment (WISE; Linn, Clark, & Slotta, 2003), that is attempting to address issues of scalability and sustainability.

Design-Based Research

Many cognitively oriented technology innovations are developed using design-based methodologies in which researchers work closely with teachers and students to design, develop, implement, and evaluate an innovation in real classroom settings (A. L. Brown, 1992; Collins, 1990). Design-based research has great potential for creating “usable knowledge” (Lagemann, 2002) principally because it is intertwined with practice and makes an attempt to study the complex influences of context on teaching and learning (The Design-Based Research Collective, 2003). Design-based research combines inductive qualitative approaches with quantitative and quasi-experimental approaches, varying the method to suit research questions that present themselves over the life of the collaboration. Research staff generally establish a regular presence in the classroom to support the use of the innovation, sometimes modeling or co-leading instruction with the teacher (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003). In part, this serves to temporarily establish conditions that are favorable to the innovation’s success. Without these conditions, it would not be possible to study the phenomena or ideas of interest. However, if the conditions depend heavily upon an infusion of extra support from researchers, this may pose a challenge to scalability and sustainability.

Another characteristic of design-based research on cognitively oriented technology innovations is the nature of the participants. As Means (1998) put it,

In the majority of cases (but not always), teachers are voluntary participants, and hence likely both to buy-in to the philosophy of the project and to see the connection of the technology used in the project to something they want to do with their students …. The disadvantage innovative technology-supported projects often face is the fact that they may not be a good match to priorities with an individual school or district. (p. 7)

Design-based research, which has been a major methodology within the Learning Sciences, has the advantage of grounding the lessons of research on cognitively oriented innovations firmly in dilemmas of practice (Shepard, 2000), but in a specialized way that does not necessarily lead to the sustainability or scalability of the innovations. Work to date, and design-based efforts in particular, have helped re searchers to better understand the constraints and contexts for classroom uses of learning technologies. But what happens when the innovation is used by dozens or hundreds of teachers who do not share co-ownership of the design with the researchers and may lack specialized knowledge generated from the collaborative research process?

Design-based research focuses on studying an innovation’s use within a classroom or several classrooms, and not necessarily on “external” factors that are necessary for the innovation’s support. Writing about methodological issues in design research, Collins, Joseph, and Bielaczyc (this issue) point to the importance of including a school or institutional analysis among the ways that a particular design can be studied. They also argue that system variables, such as ease of adoption, sustainability, and spread, are key dependent variables that should be measured in design-based research. We agree, but believe that if design-based research is going to provide guidance for systemic reform, such variables need to be treated as more than outcome measures, but as a central part of the intervention. If we are to foster truly sustainable innovations, there is a pressing need for an extension of classroom-based design research that focuses on schools and school systems as the primary units of analysis (Snipes, Doolittle, & Herlihy, 2002). As we work to build upon the lessons learned from classroom-oriented design-based research, we need to define questions that explicitly address issues of scalability and sustainability, if we hope for innovations to enter into widespread use beyond their original research contexts. To this end, we now turn our attention in this article to the constraints and needs of school systems.

SUSTAINABILITY, SCALABILITY, AND SYSTEMIC REFORM

Developers of cognitively oriented technology innovations want their innovations to be sustainable and scalable. The Learning Sciences is a field rooted in cognitive science, but with practice-oriented objectives. Ideally, the use of innovations will extend beyond the time that researchers are directly involved in the classroom. This is the essence of sustainability. Ideally, the pedagogical ideas and uses of technology that are encompassed in the innovation will spread to teachers’ general repertoires. If school systems are able to support the practices embodied in the innovation, then their use will also spread to other teachers within or across schools. This is the essence of scalability.

There are various mechanisms employed in attempts to create sustainability and scale. We have chosen to work in the context of urban systemic reform, in which an innovation is intended to reach many teachers and students within a single school system. A fundamental challenge of work in systemic reform contexts is creating alignment across the components of school systems, such as administration and management, curriculum and instruction, assessment, policy, and technology (Smith & O’Day, 1991). If the challenge of alignment can be met, an innovation has a better chance of being both sustained and scaled because the alignment of the system creates a stable structure and provides needed support. Moreover, cognitively oriented technology innovations that are well matched to systemic reform goals of school systems, such as standards-based instruction, are more likely to be sustainable, and more likely to be scalable to widespread use. These challenges are not new to th...

Table of contents

• Cover
• Half Title
• Table of Contents
• Design-Based Research: Putting a Stake in the Ground
• Design Research: Theoretical and Methodological Issues
• Creating a Framework for Research on Systemic Technology Innovations.
• Ontological Innovation and the Role of Theory in Design Experiments
• COMMENTARIES If Design-Based Research is the Answer, What is the Question? A Commentary on Collins, Joseph, and Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS Special Issue on Design-Based Research
• Design Research in Education: Yes, but is it Methodological?