The Tyranny of Metrics

Jerry Z. Muller

• Publisher: Princeton University Press
• Year: 2019
• ISBN:
  9780691191263
• Topic: Betriebswirtschaft
• Subtopic: Verwaltung

How the obsession with quantifying human performance threatens business, medicine, education, government—and the quality of our lives Today, organizations of all kinds are ruled by the belief that the path to success is quantifying human performance, publicizing the results, and dividing up the rewards based on the numbers. But in our zeal to instill the evaluation process with scientific rigor, we've gone from measuring performance to fixating on measuring itself—and this tyranny of metrics now threatens the quality of our organizations and lives. In this brief, accessible, and powerful book, Jerry Muller uncovers the damage metrics are causing and shows how we can begin to fix the problem. Filled with examples from business, medicine, education, government, and other fields, the book explains why paying for measured performance doesn't work, why surgical scorecards may increase deaths, and much more. But Muller also shows that, when used as a complement to judgment based on personal experience, metrics can be beneficial, and he includes an invaluable checklist of when and how to use them. The result is an essential corrective to a harmful trend that increasingly affects us all.

III

THE MISMEASURE OF ALL THINGS?

CASE STUDIES

7

COLLEGES AND UNIVERSITIES

Let’s take as our first case study the realm of higher education, the ground zero of my own investigations of metric fixation. Comprising a huge sector of the national economy and a central institution of all advanced societies, colleges and universities exemplify many of the characteristic flaws and unintended consequences of measured performance, as well as some of its advantages.

RAISING THE METRIC: EVERYONE SHOULD GO TO COLLEGE

Once we become fixated on measurement, we easily slip into believing that more is better.

More and more Americans are going on to post–high school education, encouraged to do so by both governments and nonprofit organizations. According to the U.S. Department of Education, for example, “In today’s world, college is not a luxury that only some Americans can afford to enjoy; it is an economic, civic, and personal necessity for all Americans.”¹

One of many nonprofit organizations that convey the same message is the Lumina Foundation. Its mission is to expand post-secondary educational attainment, with a goal of having 60 percent of Americans hold a college degree, certificate, or other “high-quality postsecondary credential” by the year 2025. Its “Stronger Nation” initiative, as the foundation declares on its website,

is all about the evidence of that learning—quantifying it, tracking it, pinpointing the places where it is and isn’t happening…. Lumina is also working with state policy leaders across the nation to set attainment goals and develop and implement strong state plans to reach them. So far, 26 states have set rigorous and challenging attainment goals—15 in the last year alone. Most of these states are taking concrete steps—such as implementing outcomes-based funding, improving developmental education, and making higher education more affordable—to increase attainment and reach their goals.²

The Lumina Foundation is steeped in metrics and proselytizes on its behalf: its website proclaims, “As an organization focused on results, Lumina Foundation uses a set of national metrics to guide our work, measure our impact and monitor the nation’s progress toward Goal 2025.”

The Lumina Foundation’s mission comports with a widely shared conviction about the role of higher education in American society: the belief that ever more people should go on to college, and that doing so increases not only their own life-time earnings but also creates national economic growth.

RAISING THE NUMBER OF WINNERS LOWERS THE VALUE OF WINNING

That article of faith, and the performance targets to which it gives rise, may simply be mistaken. As Alison Wolf, an educational economist at the University of London, has pointed out, it is true that those who have a B.A. tend to earn more on average than those without one. Thus, on the individual level, the quest for a B.A. degree may make economic sense. But on the national level, the idea that more university graduates means higher productivity is a fallacy.³

One reason for that is that to a large extent education is a positional good—at least when it comes to the job market. For potential employers, degrees act as signals: they serve as a shorthand that allows employers to rank initial applicants for a job. Having completed high school signals a certain, modest level of intellectual competence as well as personality traits such as persistence. Finishing college is a signal of a somewhat higher level of each of these. In a society where a small minority successfully completes college, having a B.A. signals a certain measure of superiority. But the higher the percentage of people with a B.A., the lower its value as a sorting device. What happens instead is that jobs that once required only a high school diploma now require a B.A. That is not because the jobs have become more cognitively demanding or require a higher level of skill, but because employers can afford to choose from among the many applicants who hold a B.A., while excluding the rest. The result is both to depress the wages of those who lack a college degree, and to place many college graduates in jobs that don’t actually make use of the substance of their college education.⁴ That leads to a positional arms race: as word spreads that a college diploma is the entry ticket to even modest jobs, more and more people seek degrees.

Thus, there are private incentives for increasing numbers of people to try to obtain a college degree. Meanwhile, governments and private organizations set performance measures aimed at raising college attendance and graduation.

HIGHER METRICS THROUGH LOWER STANDARDS

But the fact that more Americans are entering college does not mean that they are prepared to do so, or that all Americans are capable of actually earning a meaningful college degree.

In fact, there is no indication that more students are leaving high school prepared for college-level work.⁵ One measure of college preparedness is the performance of students on achievement tests, such as the SAT and the ACT, which are used to predict likely success in college (they are, in part, aptitude tests). For the most part, these tests are taken only by high school students who have some hope of going on to higher education, though in an effort to boost student achievement, some states have taken to mandating that ever more students take such tests. (Probably a case of misplaced causation. Students who took the tests tended to have higher levels of achievement. So, it was mistakenly reasoned, by getting more students to take the test, levels of achievement would be raised. The flaw is that better-performing students were more likely to take the test in the first place. That is, policymakers mistook cause for effect.) The ACT tests four subject areas: English, math, reading, and science. The company that develops the ACT has developed benchmarks of scores that indicate that the test taker has a “strong readiness for college course work.” Of those who took the ACT test most recently, a third did not meet the benchmark in any of the four categories, and only 38 percent met the benchmarks in at least three of the four areas. In short, most of those who aspire to go on to college do not have the demonstrated ability to do so.⁶

The results are predictable—though few want to acknowledge them. Since more students enter community colleges and four-year colleges inadequately prepared, a large portion require remedial courses. These are courses (now euphemistically rechristened “developmental” courses) that cover what the student ought to have learned in high school. A third of students who enter community colleges are placed in developmental reading classes, and more than 59 percent are placed in developmental mathematics courses.⁷ Students who are inadequately prepared for college also make additional demands on the institutions they attend, thus raising the costs of college education: the growth on campuses of centers of “educational excellence” is a euphemistic response to the need for more extracurricular help in writing and other skills for students inadequately prepared for university-level work.

Colleges, both public and private, are measured and rewarded based in part on their graduation rates, which are one of the criteria by which colleges are ranked, and in some cases, remunerated. (Recall the Lumina Foundation’s encouragement of state governments to engage in “outcomes-based funding.”) What then happens is that outcomes follow funding. By allowing more students to pass, a college transparently demonstrates its accountability through its excellent metric of performance. What is not so transparent is the lowered standards demanded for graduation.⁸ More courses are offered with requirements that are easily fulfilled. There is pressure on professors—sometimes overt, sometimes tacit⁹—to be generous in awarding grades. An ever-larger portion of the teaching faculty comprises adjunct instructors—and an adjunct who fails a substantial portion of her class (even if their performance merits it) is less likely to have her contract renewed.

Thus, more students are entering colleges and universities. A consequence of students entering college without the ability to do college-level work is the ever larger number of students who enroll but do not complete their degrees—a widespread and growing phenomenon that has substantial costs for the students who do so, in tuition, living expenses, and earnings foregone.¹⁰ High dropout rates seem to indicate that too many students are attempting college, not too few.¹¹ And those who do obtain degrees find that a generic B.A. is of diminishing economic value, because it signals less and less to potential employers about real ability and achievement.¹² Recognizing this, prospective college students and their parents seek admission not just to any college, but to a highly ranked one.¹³ And that, in turn, has led to the arms race of college rankings, a topic to which we will return.

Lowering the standards for obtaining a B.A. means that using the percentage of those who attain a college degree as an indicator of “human capital” becomes a deceptive unit of measurement for public policy analysis. Economists can evaluate only what they can measure, and what they can measure needs to be standardized. Thus economists who work on “human capital” and its contribution to economic growth (and who almost always conclude that what the economy needs is more college graduates) often use college graduation rates as their measure of “human capital” attainment, ignoring the fact that not all B.A.’s are the same, and that some may not reflect much ability or achievement. This lends a certain air of unreality to the explorations of what one might call the unworldly economists, who combine hard measures of statistical validity with weak interest in the validity of the units of measurement.

One assumption that lies behind the effort to boost levels of college enrollment and completion is that increases in average educational attainment somehow translate into higher levels of national economic growth. But some distinguished economists on both sides of the Atlantic—Alison Wolf in England, and Daron Acemoglu and David Autor in the United States—have concluded that that is no longer the case, if it ever was. In an age in which technology is replacing many tasks previously performed by those with low to moderate levels of human capital, national economic growth based on innovation and technological progress depends not so much on the average level of educational attainment as on the attainment of those at the top of the distribution of knowledge, ability, and skill.¹⁴ In recent decades, the percentage of the population with a college degree has gone up, while the rate of economic growth has declined. And though the gap between the earnings of those with and those without a college diploma remains substantial, the falling rate of earnings for college graduates seems to indicate that the economy already has an oversupply of graduates.¹⁵ By contrast, there is a shortage of workers in the skilled trades, such as plumbers, carpenters, and electricians—occupations in which training occurs through apprenticeship rather than through college education—who often earn more than those with four-year degrees.¹⁶

To be sure, public policy ought to aim at more than economic growth, and there is more to college education than its effect on earning capacity, as we will explore in a moment. But for now, it is worth underscoring that the metric goal of ever more college graduates is dubious even by the economistic criteria by which higher education is often measured.

PRESSURE TO MEASURE COLLEGE PERFORMANCE

In the decades since Elie Kedourie penned his critique of the centralizing policy of Margaret Thatcher’s Conservative government, central government control over British institutions of higher education has expanded and intensified. Much of that control takes the form of management through performance metrics. For scholarship in many fields, the results have been deleterious.

In England, as elsewhere, an ever larger proportion of the population is attending university, in keeping with the government’s aims. In 1970 less than 10 percent of men and women in each age cohort attended university. By 1997, it was close to a third, and by 2012, 38 percent of nineteen-year-olds were enrolled in some form of tertiary education.¹⁷ Paying for them is an ever more onerous task, and in recent years the costs have been increasingly shifted to the students themselves (or their families) in the form of tuition fees. But government expenditure remains substantial, and in an effort to control expenses and achieve “value,” that control increasingly takes the form of payment for purported results. That performance is evaluated through metrics that focus upon the measured output of each department and institution.

In an attempt to obtain “value,” successive British administrations have created a series of government agencies charged with evaluating the country’s universities, with titles such as the “Quality Assurance Agency.”¹⁸ There are audits of teaching quality, such as the “Teaching Quality Assessment,” evaluated largely on the extent to which various procedures are followed and paperwork filed, few of which have much to do with actual teaching.¹⁹ But one clear result has been that professors are forced to devote more and more of their time to paperwork rather than to research or teaching. And there has been a ballooning of the number of professional staff, including the newly created post of “quality assurance officers,” dedicated to gathering and analyzing the data for what was once known as the Research Assessment Exercise, since rechristened as the Research Excellence Framework.²⁰ The cost of these exercises in metrics in England alone was estimated at £250,000,000 in 2002.²¹ A mushroom-like growth of administrative staff has occurred in other countries that have adopted similar systems of performance measurement, such as Australia. In most such systems, metrics has diverted time and resources away from doing and toward documenting, and from those who teach and research to those who gather and disseminate the data for the Research Assessment Exercise and its counterparts.²² The search for more data means more data managers, more bureaucracy, more expensive software systems. Ironically, in the name of controlling costs, expenditures wax.

The closest parallel in the United States are the accrediting organizations that grant legitimacy to American colleges and universities. They are regional in scope, but since receiving federal funds requires accreditation by such agencies, they also serve as instruments of the federal government.²³ While they do not contro...