Using Quality Benchmarks for Assessing and Developing Undergraduate Programs

Dana S. Dunn, Maureen A. McCarthy, Suzanne C. Baker, Jane S. Halonen

• Publisher: Jossey-Bass
• Year: 2010
• Print ISBN: 9780470405567
• eBook ISBN: 9780470892473
• Edition: 1
• Topic: Education
• Subtopic: Higher Education

Using Quality Benchmarks for Assessing and Developing Undergraduate Programs

Using Quality Benchmarks for Assessing and Developing Undergraduate Programs introduces selected performance criteria—benchmarks—to assist undergraduate programs in defining their educational goals and documenting their effectiveness.

The book explores the attributes of undergraduate programs by focusing on educationally related activities in eight domains: program climate; assessment, accountability, and accreditation issues; student learning outcomes; student development; curriculum; faculty characteristics; program resources; and administrative support. Further, it conceptualizes a continuum of performance for each attribute in each of the domains to characterize underdeveloped, developing, effective, and distinguished achievement for undergraduate programs.

The goal of the book is to encourage individual departments at various types of institutions to evaluate what they currently do well while identifying areas for refinement or future growth. When benchmarks reveal that a program is underdeveloped, faculty and administrators can plan for how they can best direct subsequent efforts and resources to improve a program's performance and ability to serve students.

Emphasizing formative assessment over summative or punitive evaluation, the benchmarks in this book are designed to improve program quality, encourage more effective program reviews, and help optimally functioning programs compete more successfully for resources. Using performance benchmarks to identify areas of program strength can, in turn, be used to recruit and retain students, seek funding via grants or alumni support, and enhance the perceived rating of an institution.

Chapter 1

The Need for Quality Benchmarks in Undergraduate Programs

The hour for recognizing the singular importance of undergraduate education is here at last. Department chairs, faculty at all ranks, deans and provosts, and even college and university presidents finally realize that educating undergraduates is arguably the most important mission of higher education today. For those of us who have watched the focus shift from graduate and post-baccalaureate professional education to the education of traditional and nontraditional four-year students, the change is both powerful and palpable. Teaching undergraduates well is now a dominant focus in higher education. We are not suggesting that educators and administrators should not attend to the important and pressing concerns of graduate education; rather, we believe that the best support graduate and professional schools can receive is to send them well-prepared students.

As a result of the new emphasis on undergraduate education, colleges and universities aspire to provide the best educational experience for student majors that their resources will permit. Their common goal is to assess what and how well students have learned during their time at the institution. The issues involved, similar to the work itself, are challenging. Where should institutional self-reflection about undergraduate learning begin? Within the administration? Among the collective faculty? Or should alumni be tapped for their feedback on how their time at the institution shaped their futures? What about the opinions of other stakeholders, including the students themselves?

When institutions turn their focus to undergraduate education, we argue that the place to begin this important work is at the departmental or program level—the unit of analysis that has the most day-to-day as well as discipline-based impact on student learning. Assessment in undergraduate education is often aimed at general education; that is, the distribution requirements of liberal arts offerings that all students in a given college or university must complete. Evaluating student learning in general education courses is certainly important, but we believe that the breadth and depth of discipline-based knowledge acquired within department-based majors is the more appropriate forum to capture assessment activities that reflect the true accomplishments of the baccalaureate program. How well are chemistry majors learning foundational materials in basic and intermediate courses? Does this foundational material later help these student majors display the necessary critical thinking skills in the advanced courses in the major? As they near the end of their major courses, can chemistry students conceive and design experiments? Are they able to interpret and explain the results of their research using the discipline's vernacular? The same sorts of questions could be appropriately framed and examined for any other major area of study, whether it be architecture, urban planning, or Urdu.

One challenge is that there is little formal consensus about what constitutes program quality in undergraduate education in the arts and humanities, social sciences, and natural sciences. Ratings or rankings are either aimed at graduate programs (for example, the top twenty forestry programs in the United States, best graduate business schools on the West coast) or, as we will see, the placement of the institution as a whole on higher education's pantheon (for example, the list of best regional small universities in the southeast, the top one hundred national liberal arts colleges). Ratings and rankings have their place, but they tell a limited story regarding quality undergraduate education. We wrote this book to help educators at all sizes and types of four-year institutions, including teachers, researchers, scholars and administrators, to constructively evaluate and document the effectiveness of current programs aimed at educating undergraduates.

We realize that there is nothing new about conducting academic program reviews. As is well known, most colleges and universities have implemented a formal review cycle for individual departments or programs, typically every five years or so. The pursuit of “educational quality,” for example, has been a key campus concern for a couple of decades now (see Berquist & Armstrong, 1986; Bogue & Saunders, 1992; Boyer, 1987; Haworth & Conrad, 1997). Following the groundbreaking work of Boyer (1990), debates about the nature of scholarship have become standard procedure as campuses define the implications for what faculty should do in and out of the classroom. What is new, however, is the availability of helpful assessment tools for evaluating the strengths, challenges, and unrealized opportunities within departments or other programs. Such tools—including discipline-based learning guidelines, curriculum evaluation guides, standardized student surveys, rubrics for teaching and evaluating writing and speaking demonstrations, and instructor-designed measures tailored to evaluate whether the learning goals of assignments are met—constitute some other possibilities.

We advocate the use of a particular assessment tool: quality benchmarks; that is, reasonable, reason-based, and peer-sanctioned criteria that can be used to assess the performance of academic programs and departments. Benchmarks provide a guiding standard for comparing what is with what could be achieved with redirected effort, energy, attention, or resources (see, for example, Astin, 1993a; Banta, Lund, Black, & Oblander, 1996). Formative rather than summative, quality benchmarks allow constituents—department chairs, program directors, faculty, deans, and students—to review progress, identify problems, establish or revisit goals, and reflect on student learning outcomes or establish new ones.

The term “benchmarks” has a particular meaning that differs somewhat from terms such as “criteria” or “rubrics” and the like. Benchmarking is the process of assessing a program's qualities by comparing them with—and trying to shape them to conform to—what are considered to be the best practices or ideal program features. In essence, benchmarking provides a snapshot of program performance that helps the relevant constituents understand how particular educational activities within key domains compare with acknowledged standards. The upshot of such benchmarking is that a program will undertake changes in order to improve teaching and learning. The aim of benchmarking is to improve some particular aspect of program performance (such as student research skills, faculty scholarly output), but it is part of a larger continuous process in which departments and the colleagues who teach and administrate within them continually seek to challenge their practices.

We argue that benchmarks provide a set of standard reference points for internal self-studies and external reviews. Of course, benchmarking processes also are routinely invoked when comparing one program to another or when sizing up whether a program has achieved the standards set forth by an accrediting agency. Regardless of whether the focus involves internal or external standards, the process of comparing achievement to an identified standard can serve multiple purposes: benchmarking can assess institutional reputation locally or nationally, verify goal achievements, and measure whether outcome targets have been realized, among other objectives.

The emphasis of the benchmarks we propose is internal. We suggest using these selected quality benchmarks to assist undergraduate programs in establishing quality objectives, monitoring progress toward their achievement, and ultimately achieving a level commensurate with department talent, energy, and resources. In the process, working with developmental standards can further define their program missions and document their effectiveness.

We also want to be clear that benchmarks are not national standards for accrediting departments or programs. We are not proposing that the quality benchmarks presented in this book should be used to compare a given institution with its peer or aspirant counterparts. Again, we propose that departments and other academic programs establish appropriate internal benchmarks for measuring their own progress on various dimensions. By doing so, we advocate that departments concentrate on their own goal setting and progress. Further, accreditation standards are generally all-or-nothing propositions, whereas quality benchmarks are designed to be developmental measures for assessing a program. We urge readers to keep these points in mind as they read this book and subsequently reflect on, evaluate, and improve their departments or programs.

An Example: Using Benchmarks for Program Advocacy

Consider a familiar example on many campuses. A department chair wants to search for and hire a new faculty member to offer courses in a key topical area that currently receives little or no coverage in the program's curriculum. The dean or the vice president for academic affairs (VPAA) reviews the request and suggests that a self-study is in order before any additional faculty line can be considered. The dean also points out that several other departments are simultaneously claiming an equally pressing need for new hires but the projected budget for the next academic year cannot possibly accommodate all requests. To make things interesting, let's imagine that enrollment in the department is solid and steady, but some of the competing departments have seen moderate growth in the numbers of students they serve.

How can this department chair hope to convince the dean that the requested hire is truly needed and worthy of support? How can the chair convince herself that her request is justified, given the needs of her colleagues who are leading other departments? We'll present two possible scenarios.

Scenario one. The chair dutifully undertakes the writing of a self-study, the first in several years. Department colleagues are supportive but expect her to carry the burden of gathering information, drafting arguments for the position, and writing the actual proposal. The majority of department members agree that the hire is justified, but aside from drawing up a list of reasons for the hire from the department's own perspective and perceived immediate needs (for example, enrollment crunch, recent retirements, most recent new line was five years ago), little attention is paid to how the hire could fit into the larger constellation of department issues, including curricular changes, wider institutional service, and enhancing undergraduate education. In other words, the department operates in an academic “business as usual” model, coupled with the oft-heard claim of “too many students to serve, not enough resources.”

Although the final proposal was compelling in many respects, the dean received equally cogent arguments from other departments. In the end, the dean gave the hire to another department, one with fewer full-time faculty members than the others and, up to that point, larger class sizes. After spending considerable time and effort on the proposal, the department chair is somewhat demoralized; her colleagues are frustrated, disappointed, and somewhat aggrieved. What little collaborative spirit existed in the department disappears for quite some time.

Scenario two. Imagine that the perceived need to hire a new colleague spurs the interest and involvement of the entire department. They agree to an approach based on benchmarking. Instead of expecting the chairperson to do all the work, the faculty members divide up the tasks related to the potential hire. For example, some colleagues identify both new and established courses that the new person would teach (curriculum), others examine the hire's advising responsibilities (student development), and so on. With the help of colleagues, the chair drafts a proposal that documents the department's current mission and goals, highlighting particular areas that are distinguished (for example, a recently revised curriculum that meets national disciplinary standards, an outreach program to the local community) as well as those that are effective (such as undergraduate research presentations delivered at regional and national conferences), or still developing (for example, the department now requires that students give formal, oral presentations in upper-level courses), or that need attention because they are undeveloped (for example, faculty publishing has declined precipitously in recent years). (We define and explain the italicized terms in the next section of this chapter.)

The proposal specifically explains how a new hire with a desired specialty can contribute to the areas of strength while also supporting the developing and underdeveloped areas of the department. In the course of the self-study, members of the department streamlined a few procedural issues that ended up helping the department's budget. The comprehensive nature of the proposal clearly documented the need for the new hire. The department chair was gratified by the level of enthusiasm, camaraderie, and participation of her colleagues. The colleagues, in turn, realized that while there were some areas of concern, the department was clearly moving in the right direction. The dean concurred, impressed by the amount of effort, goodwill, and careful, thoughtful planning. The proposal was much more focused and reason-based than the competing proposals from the other departments.

Scenario one is all too familiar on most campuses. Scenario two is less familiar. Let's imagine that the dean in scenario two is convinced by the evidence and grants the new faculty line. Everyone in the department is overjoyed, especially the chair. But consider this: even if such benchmarking does not always succeed—there are usually any number of legitimate reasons to postpone valid hires—the act of comparing the current state against quality performance benchmarks still provides department members, the chair, and the dean with a sense of which aspects of the program are working well (often very well) and where some improvements could be made. Most important, not all improvements require an outlay of capital or an expanded budget; some are procedural, others organizational, and some rely on a combination of good will and common sense. The point is that something beneficial grew from the program review that embraced the benchmarking approach.

What are some of the concrete advantages of using performance benchmarks? These include:

• Engagement of faculty and students in crafting and revising the mission statement of a department or program
• Formative evaluation of teaching and learning outcomes
• Curricular review, refinement, and revision
• Recruitment and retention of quality faculty and students
• Assessment of resource needs
• Long-range academic program planning
• Evaluation and demonstration of program quality

Benchmarking and Program Assessment for Educational Renewal

Benchmarks are used in higher education as a way to improve the climate for learning within departments (see, for example, Umbach & Porter, 2002; Umbach & Wawrynski, 2005). We recently proposed performance benchmarks to assist undergraduate psychology programs in defining their missions and goals as well as in documenting their effectiveness (Dunn, McCarthy, Baker, Halonen, & Hill, 2007). The developmental framework we proposed garnered considerable interest among communities of teachers and administrators within the discipline of psychology. The interest we received led us to think more broadly about how quality benchmarks could be used to evaluate virtually any academic department or program. Indeed, our experiences as program reviewers, faculty members, and part- or full-time administrators informs us that departments, department heads, and deans all want reasonable, reliable, and professionally respectable methods for evaluating the performance of undergraduate programs.

We believe the developmental framework presented in this book will satisfy all parties because we emphasize formative assessment. We do not advocate the use of summative assessment—here defined as the use of benchmarks to reward (for example, add faculty lines, build new facilities) or punish (for example, eliminate faculty, trim the budget) a program for its standing. Furthermore, we designed our framework to help programs identify and tout what they already do well even in situations involving seriously constrained resources. Finally, the performance benchmarks used to identify areas of program strength can, in turn, be used to recruit and retain students, to seek funding via grants or alumni support, and to enhance the perceived rating of an institution. When benchmarks reveal that a program or areas within it are undeveloped or developing, faculty and administrators can then plan where subsequent efforts and resources are best placed to raise a program's standing.

Our framework explores the attributes of undergraduate programs by focusing on educationally related activities in eight domains: program climate; assessment, accountability, and accreditation issues; student learning outcomes; student development; curriculum; faculty characteristics; program resources; and administrative support. We conceptualize a continuum of performance for each attribute in each of the domains to characterize undeveloped, developing, effective, and distinguished achievement for undergraduate programs. We will discuss this continuum of performance in more detail shortly. Our goal is to encourage individual departments at various types of institutions to evaluate what they currently do formatively while identifying areas for refinement or future growth. We believe that our recommended benchmarks can improve program quality, encourage more effective program reviews, and help optimally functioning programs compete more successfully for resources based on their distinguished achievements.

Characterizing Program Performance

Within any of the eight educational domains, we construed a program's performance attributes as characterized along a fourfold continuum from undeveloped to distinguished. Exhibit 1.1 lists and defines the continuum's characteristics. An undeveloped characteristic is one that is interfering with a department or program's ability to pursue its educational mission...

Table of contents

• Cover
• Title Page
• Copyright
• Dedication
• Foreword
• Preface
• About the Authors
• Chapter 1: The Need for Quality Benchmarks in Undergraduate Programs
• Part One: Benchmarking for Eight Key Program Domains
• Part Two: Benchmarking in Practice
• Appendix A: Using Benchmarking to Serve as an External Reviewer
• Appendix B: Sources of Data
• Appendix C: Assessment Materials for the Arts
• Appendix D: Disciplinary Accrediting Organizations for Bachelor's Degree Programs Currently Recognized by the Council for Higher Education Accreditation
• References
• Index