System Error

How Big Tech Disrupted Everything and Why We Must Reboot

Rob Reich, Mehran Sahami, Jeremy M. Weinstein

• Publisher: Harper
• Year: 2021
• ISBN:
  9780063066205
• Topic: Business
• Subtopic: IT Industry
• Index: Business

" System Error is a triumph: an analysis of the critical challenges facing our digital society that is as accessible as it is sophisticated." —Anne-Marie Slaughter, CEO of New America

A forward-thinking manifesto from three Stanford professors—experts who have worked at ground zero of the tech revolution for decades—which reveals how big tech's obsession with optimization and efficiency has sacrificed fundamental human values and outlines steps we can take to change course, renew our democracy, and save ourselves.

In no more than the blink of an eye, a naïve optimism about technology's liberating potential has given way to a dystopian obsession with biased algorithms, surveillance capitalism, and job-displacing robots. Yet too few of us see any alternative to accepting the onward march of technology. We have simply accepted a technological future designed for us by technologists, the venture capitalists who fund them, and the politicians who give them free rein.

It doesn't need to be this way.

System Error exposes the root of our current predicament: how big tech's relentless focus on optimization is driving a future that reinforces discrimination, erodes privacy, displaces workers, and pollutes the information we get. This optimization mindset substitutes what companies care about for the values that we as a democratic society might choose to prioritize. Well-intentioned optimizers fail to measure all that is meaningful and, when their creative disruptions achieve great scale, they impose their values upon the rest of us.

Armed with an understanding of how technologists think and exercise their power, three Stanford professors—a philosopher working at the intersection of tech and ethics, a political scientist who served under Obama, and the director of the undergraduate Computer Science program at Stanford (also an early Google engineer)—reveal how we can hold that power to account.

Troubled by the values that permeate the university's student body and its culture, they worked together to chart a new path forward, creating a popular course to transform how tomorrow's technologists approach their profession. Now, as the dominance of big tech becomes an explosive societal conundrum, they share their provocative insights and concrete solutions to help everyone understand what is happening, what is at stake, and what we can do to control technology instead of letting it control us.

Part I

Decoding the Technologists

In an age of advanced technology, inefficiency is the sin against the Holy Ghost.

—Aldous Huxley, foreword to Brave New World, 1946

Chapter 1

The Imperfections of the Optimization Mindset

Contrary to current popular opinion, for most of its storied existence the United States Postal Service has been a hub of disruptive innovation. In 1792, Benjamin Rush, a signer of the Declaration of Independence, ushered the Postal Service Act into law, giving the federal government control over various regional postal routes and declaring that the content of mail was private, even if delivered by a public carrier. It also provided for a novel cost structure: rather than charging on the basis of weight, the postal service would charge one rate for letters and another much less expensive rate for newspapers. The exchange and broad dissemination of news and information would thus be subsidized.

In order to improve the efficiency and reliability of mail delivery throughout a quickly expanding nation, the post office regularly turned to new technologies. It introduced delivery by horse, the Pony Express, in 1862 and began experimenting with delivery by train a few years later. To connect as many people as possible, the post office introduced free delivery of mail to rural areas in 1902. And just a few years after Henry Ford invented the automobile in 1901 and the Wright brothers the airplane in 1903, the post office was trying out both technologies for mail delivery.

In 1913, during the height of railroad development and the dawn of the age of the automobile, the post office introduced another radical and innovative idea: parcel post, designed to facilitate the delivery of ordinary merchandise via mail. In the expanding age of commerce, the post office would deliver goods in addition to letters, newspapers, and magazines. New forms of business sprang up to take advantage of this new delivery mechanism. Mail-order companies boomed. With thousands of carriers spread across the country, including in rural areas, mail-order industry profits went from $40 million in 1908 to $250 million in 1920, a huge sum at the time.

Imagine a family in 1920 in a rural area of the United States that is planning a trip and needs to buy a bunch of supplies. Rather than having to go many miles to the nearest city by horse or train, the family can order by mail from the Sears, Roebuck catalogue, choosing from an array of goods much larger than any shop in their own town is likely to stock. After receiving the order, Sears, Roebuck will fulfill it using parcel post. The process from start to finish might take a few weeks. That is still a long time but much more efficient and convenient than the alternatives.

Fast-forward a century. The post office is facing a crisis in the digital age. Digital technologies have driven greater and greater efficiencies in the exchange of messages. Email and text messaging now permit the instantaneous delivery of personal messages across the greatest of distances. The volume of single letters sent by first-class mail declined by 61 percent from 1995 to 2013. With the ubiquity of email and text messaging, writing a letter has become more and more a rarity, an expression of some romantic inclination rather than the humdrum desire to communicate or get something done. Ordering merchandise from catalogues via mail still exists but has declined precipitously. The Sears, Roebuck catalogue folded in 1993, and the company filed for bankruptcy in 2018.

How would that family in a rural area outfit themselves for a trip today? They would likely order what they need from Amazon Prime with the click of a few buttons, and in two days the packages would be delivered, possibly by the post office or a private carrier such as FedEx. In some cities, the packages might even arrive in two hours with Amazon Prime Now, bypassing the post office altogether. And Amazon Prime Air promises the future possibility of delivery by drones in thirty minutes or less. A family could shop in the morning and be ready to head out with all their supplies in the afternoon.

What a story of progress through greater and greater efficiency! Instantaneous mail! Delivery of packages within a few days, hours, or even minutes!

The evolution of Netflix tells a similar story about the drive for efficiency. At its start, Netflix was a subscription service for movies, not unlike Blockbuster, the bricks-and-mortar behemoth of the movie rental business. For a monthly fee, Netflix would deliver DVDs using the US Postal Service, providing a custom-designed, instantly recognizable red postage-paid envelope for returning them. In the company’s early days, customer satisfaction hinged on there being as quick a turnaround time as possible between the return of a DVD and the arrival in the mail of the next movie in the customer’s queue. If customers had to wait a week for a new movie to arrive, they complained. Although Netflix made extraordinary efforts to make the turnaround time as short as possible, it wasn’t responsible for carrying the red envelopes in the mail; that was the post office’s job. Netflix provided customized machines in thousands of post offices to speed up the return process, aiming for one-day delivery to its subscribers. And at one point, looking for every possible means to improve on delivery and return times, it hired the former US postmaster general to be its chief operations officer. Who better to improve Netflix’s ability to navigate the postal system?

But even in its early days, Netflix’s long-term plan wasn’t just about mailing DVDs. Reed Hastings, the company’s founder and CEO, who holds a master’s degree in computer science, knew that it was only a matter of time until efficiency gains in internet communication would enable the streaming of video directly to consumers. In 2007, his vision came to fruition. Taking advantage of broadband access, Netflix could bypass the postal service altogether. The company, recognizing that it still needed to support customers with limited or no internet access, also spun out a separate service called Qwikster that would continue DVD delivery by mail. Quickly it pivoted almost entirely to delivering films and television shows on demand via streaming video. Now it’s possible to watch nearly any movie one wants to at any moment, instantaneously, via the internet. Blockbuster—which in 2000 had turned down an offer to acquire Netflix for $50 million—filed for bankruptcy in 2010.

Today the delivery of movies is not merely efficient; it has been optimized. In terms of time spent, there appear to be no possible further improvements after instantaneous delivery via streaming.

Efficiency gains have led not only to greater convenience but to outcomes that matter far more, including many that enhance democracy or economic opportunity: better distribution of essential medicines around the globe, the development of new vaccines, easier access to the world’s information, and more effective interventions for students with learning differences.

Over the past several decades, the drive toward efficiency and optimization has come to play an increasingly dominant role across spheres and industries, from business (e.g., streamlining supply chains) to sports (e.g., Moneyball-style tactics using big-data analytics to drive decisions) and even to our personal lives (e.g., online dating apps and fitness trackers). It’s no coincidence that the industry and skill set most ascendant over the same time period has been computer science. In the digital age, the disruptive innovators tend to be the efficiency-obsessed tribe of people called coders, or software engineers. They are the ones who invent and bring to market the host of new technologies that are driving efficiency gains across so many aspects of life.

Should We Optimize Everything?

Efficiency is not always the good thing it seems to be. Consider the creation of a product called Soylent, another Silicon Valley innovation driven by engineers.

Soylent is a nutritional powder that can be made into a drinkable shake by adding water. This meal replacement product was developed because its inventor believed that food is a pain point in our daily lives, an inefficient delivery mechanism for the human body’s nutritional needs. Eating food is costly and requires shopping, cooking, and cleaning up or else going out to restaurants. And many meals are social affairs, with attendant expectations of conversation and social etiquette. All of this takes considerable time away from other potentially valuable activities, such as working.

Soylent is the brainchild of Rob Rhinehart, a Silicon Valley engineer who set out to solve a specific problem. After working at a failed start-up in San Francisco, he and his friends found themselves running short on cash and facing the challenge of making a decent meal. “I started wondering why something as simple and important as food was still so inefficient, given how streamlined and optimized other modern things are,” he told Vice. As he wrote on his personal blog, “In my own life I resented the time, money, and effort the purchase, preparation, consumption, and clean-up of food was consuming. I am pretty young, generally in good health, and remain physically and mentally active. I don’t want to lose weight. I want to maintain it and spend less energy getting energy.”

So he did what engineers do: he took an engineering approach to food and nutrition. He researched the vitamins and nutrients needed for bodily sustenance. He studied materials from the Food and Drug Administration and textbooks on nutrition and biochemistry, and he made a list of more than thirty nutrients required by the human body. He ordered the nutrients over the internet and began experimenting by blending them together in a powder form.

In 2013, he started living on the powder, which he would blend into a shake, making adjustments along the way when he realized that his concoction lacked iron and that he had miscalculated the amount of fiber he needed to feel healthy. After a month of subsisting on his powdered solution alone, he wrote a blog post entitled “How I Stopped Eating Food.”

Rhinehart called his invention Soylent, taking his inspiration from a 1966 novel depicting an overpopulated world with dwindling resources in which a new food, made from soy and lentils (soy + lent), is created in order to feed people. For most folks, however, the name evokes the 1973 film adaptation of the novel, Soylent Green starring Charlton Heston. In the movie, people live on a wafer they believe to be made from plankton. The wafer is revealed in the final scenes to be produced from human flesh, and the dystopia of overpopulation turns out to be an even greater horror in which cannibalism is the only way to survive. Rhinehart never claimed to be a branding genius.

Despite this, his blog post attracted attention. It was especially popular on a site called Hacker News, a place for the tech community to learn about clever inventions and gizmos to make life better and save time. Rhinehart saw an entrepreneurial opportunity, and he posted about Soylent on a crowdfunding site, promising to deliver a week’s worth of Soylent in return for a modest donation of $65. He hoped to raise $100,000 to bootstrap production. The response was enormous. He hit his target in just two hours. Eventually, more than six thousand people sent money to support his new venture, raising a total of more than $750,000.

Rhinehart and his collaborators went into business, and in 2014 they introduced Soylent to the public. The company is funded by some of Silicon Valley’s most prominent venture capital firms. In an introductory video posted on YouTube, Rhinehart explained, “What I really learned was how to break problems down. Everything is made of parts, everything can be broken down. Unlike most other foods which prioritize taste and texture, Soylent was engineered to maximize nutrition; to nourish the body in the most efficient way possible.”

It wasn’t just the desire to maximize his nutritional needs that motivated him. He told a reporter that farms where food is grown and animals are cultivated are “very inefficient factories.” “It’s really the labor that gets me,” he said. “Agriculture’s one of the most dangerous and dirty jobs out there, and it’s traditionally done by the underclass. There’s so much walking and manual labor, counting and measuring. Surely it should be automated.”

Soylent, he said, would solve multiple problems: the inefficiency of feeding oneself with food, the stress of worrying about optimal nutrition, the food industry’s reliance on farms. Soylent would deliver all that at relatively low cost. Win-win-win-win.

The press covered the release of Soylent as “The End of Food.” In the New York Times, Farhad Manjoo complained about its “stultifying utilitarianism,” calling it a “punishingly boring, joyless product.” Soylent may offer complete nourishment but only, the tech columnist wrote, “at the expense of the aesthetic and emotional pleasures many of us crave in food.”

The newspaper assigned Sam Sifton, its food critic and restaurant reviewer, to give Soylent a try. The results were predictable: “Imagine a meal made of the milk left in the bottom of a bowl of cut-rate cereal, the liquid thickened with sweepings from the floor of a health food store, and you have some sense of what it is like to consume the protein-packed shakes that have replaced Flamin’ Hot Cheetos and Red Bull in the diets of some tech workers in Silicon Valley.” Summoning up a small bit of charity, he concluded, “These instant meals are meant for work warriors for whom good and delicious food is secondary to perfect and unassailable engineering.”

It’s not hard to spot some problems with Soylent. It may be a hyper-efficient means of meeting one’s daily nutritional needs while drastically cutting down on the time required to prepare and eat regular food. But for most people, food is not just a delivery mechanism for one’s nutritional requirements. Food serves many different ends. It brings gustatory pleasure. It provides social connection. It sustains and transmits cultural identity. A world in which Soylent spells the end of food also spells the loss of these values.

Maybe you don’t care about Soylent; it’s just another product in the marketplace that no one is required to buy. If tech workers want to economize on time spent grocery shopping or a busy person faces the choice between grabbing an unhealthy meal at a fast-food joint or bringing along some Soylent, why should anyone complain? In fact, it’s a welcome alternative for some people.

That’s fine. Engineering has obviously brought humanity lots of good. But the story of Soylent is powerful because it reveals the optimization mindset of the technologist. And problems arise when this mindset begins to dominate—when the technologies begin to scale and become universal and unavoidable.

The Education of an Engineer

In 1936, John Maynard Keynes, one of the most influential economists ever to live, observed the following:

The ideas of economists and political philosophers, both when they are right and when they are wrong, are more powerful than is commonly understood. Indeed the world is ruled by little else. Practical men, who believe themselves to be quite exempt from any intellectual influence, are usually the slaves of some defunct economist. Madmen in authority, who hear voices in the air, are distilling their frenzy from some academic scribbler of a few years back.

He wrote those words in the early part of the twentieth century, at a moment of extreme global upheaval, with the Great Depression giving way to a second world war. One might not have thought that ideas were what mattered most at such a moment. But he was right. The perspectives of economists and the contest of political ideologies served to shape the two world wars, the Cold War, the fall of the Berlin Wall, the rise of the financial sector, and a globalizing economy—virtually all of the greatest challenges of the twentieth century.

Economists also entered the innermost halls of political decision-making, advising leaders and directly crafting public policy. Prior to World War II, it was lawyers who dominated federal agencies, and courts ignored most economic evidence about the predicted effects of their decisions. But economists flooded into public service in the mid–twentieth century, growing their ranks from about two thousand in the federal government in the 1950s to more than six thousand in the 1970s. They were hired in droves by large companies to drive growth and provide economic forecasts. And the leaders of the booming Wall Street banks, private equity firms, and hedge funds that rose to prominence in the last quarter of the century all had a background in economics.

What economics and finance were to the twentieth century, engineering and computer science are to the twenty-first. Computer hardware, processing power, big data, algorithms, artificial intelligence (AI), and network power are the most important currencies of our age. The quants and financial engineers have invaded the big banks, and it is the venture capitalists of Palo Alto, not fund managers on Wall Street, who finance disruptive innovation. Yet the worldview of the technologist is sometimes poorly understood by those outside the tech industry. Unlike economists in the twentieth century, engineers are generally not entering politics as advisers and decision makers. They tend instead to bypass politics altogether.

In a...