The Black Box Society

The Secret Algorithms That Control Money and Information

Frank Pasquale

• Publisher: Harvard University Press
• Year: 2015
• Print ISBN:
  9780674970847
  9780674368279
• eBook ISBN:
  9780674967106
• Topic: Law
• Subtopic: Science & Technology Law
• Index: Law

Every day, corporations are connecting the dots about our personal behavior—silently scrutinizing clues left behind by our work habits and Internet use. The data compiled and portraits created are incredibly detailed, to the point of being invasive. But who connects the dots about what firms are doing with this information? The Black Box Society argues that we all need to be able to do so—and to set limits on how big data affects our lives.

Hidden algorithms can make (or ruin) reputations, decide the destiny of entrepreneurs, or even devastate an entire economy. Shrouded in secrecy and complexity, decisions at major Silicon Valley and Wall Street firms were long assumed to be neutral and technical. But leaks, whistleblowers, and legal disputes have shed new light on automated judgment. Self-serving and reckless behavior is surprisingly common, and easy to hide in code protected by legal and real secrecy. Even after billions of dollars of fines have been levied, underfunded regulators may have only scratched the surface of this troubling behavior.

Frank Pasquale exposes how powerful interests abuse secrecy for profit and explains ways to rein them in. Demanding transparency is only the first step. An intelligible society would assure that key decisions of its most important firms are fair, nondiscriminatory, and open to criticism. Silicon Valley and Wall Street need to accept as much accountability as they impose on others.

1

INTRODUCTION—THE NEED TO KNOW

EVERYBODY KNOWS the story about the man crawling intently around a lamppost on a dark night. When a police officer comes along and wants to know what he’s doing, he says he’s looking for his keys. “You lost them here?” asks the cop. “No,” the seeker replies, “but this is where the light is.” This bromide about futility has lately taken on a whole new meaning as a metaphor for our increasingly enigmatic technologies.

There’s a noble tradition among social scientists of trying to clarify how power works: who gets what, when, where, and why.¹ Our common life is explored in books like The Achieving Society, The Winner-Take-All Society, The Good Society, and The Decent Society. At their best, these works also tell us why such inquiry matters.²

But efforts like these are only as good as the information available. We cannot understand, or even investigate, a subject about which nothing is known. Amateur epistemologists have many names for this problem. “Unknown unknowns,” “black swans,” and “deep secrets” are popular catchphrases for our many areas of social blankness.³ There is even an emerging field of “agnotology” that studies the “structural production of ignorance, its diverse causes and conformations, whether brought about by neglect, forgetfulness, myopia, extinction, secrecy, or suppression.”⁴

Gaps in knowledge, putative and real, have powerful implications, as do the uses that are made of them. Alan Greenspan, once the most powerful central banker in the world, claimed that today’s markets are driven by an “unredeemably opaque” version of Adam Smith’s “invisible hand,” and that no one (including regulators) can ever get “more than a glimpse at the internal workings of the simplest of modern financial systems.” If this is true, libertarian policy would seem to be the only reasonable response. Friedrich von Hayek, a preeminent theorist of laissez-faire, called the “knowledge problem” an insuperable barrier to benevolent government interventions in the economy.⁵

But what if the “knowledge problem” is not an intrinsic aspect of the market, but rather is deliberately encouraged by certain businesses? What if financiers keep their doings opaque on purpose, precisely to avoid or to confound regulation? That would imply something very different about the merits of deregulation.

The challenge of the “knowledge problem” is just one example of a general truth: What we do and don’t know about the social (as opposed to the natural) world is not inherent in its nature, but is itself a function of social constructs. Much of what we can find out about companies, governments, or even one another, is governed by law. Laws of privacy, trade secrecy, the so-called Freedom of Information Act—all set limits to inquiry. They rule certain investigations out of the question before they can even begin. We need to ask: To whose benefit?

Some of these laws are crucial to a decent society. No one wants to live in a world where the boss can tape our bathroom breaks. But the laws of information protect much more than personal privacy. They allow pharmaceutical firms to hide the dangers of a new drug behind veils of trade secrecy and banks to obscure tax liabilities behind shell corporations. And they are much too valuable to their beneficiaries to be relinquished readily.

Even our political and legal systems, the spaces of our common life that are supposed to be the most open and transparent, are being colonized by the logic of secrecy. The executive branch has been lobbying ever more forcefully for the right to enact and enforce “secret law” in its pursuit of the “war on terror,” and voters contend in an electoral arena flooded with “dark money”—dollars whose donors, and whose influence, will be disclosed only after the election, if at all.⁶

But while powerful businesses, financial institutions, and government agencies hide their actions behind nondisclosure agreements, “proprietary methods,” and gag rules, our own lives are increasingly open books. Everything we do online is recorded; the only questions left are to whom the data will be available, and for how long. Anonymizing software may shield us for a little while, but who knows whether trying to hide isn’t itself the ultimate red flag for watchful authorities? Surveillance cameras, data brokers, sensor networks, and “supercookies” record how fast we drive, what pills we take, what books we read, what websites we visit. The law, so aggressively protective of secrecy in the world of commerce, is increasingly silent when it comes to the privacy of persons.

That incongruity is the focus of this book. How has secrecy become so important to industries ranging from Wall Street to Silicon Valley? What are the social implications of the invisible practices that hide the way people and businesses are labeled and treated? How can the law be used to enact the best possible balance between privacy and openness? To answer these questions is to chart a path toward a more intelligible social order.

But first, we must fully understand the problem. The term “black box” is a useful metaphor for doing so, given its own dual meaning. It can refer to a recording device, like the data-monitoring systems in planes, trains, and cars. Or it can mean a system whose workings are mysterious; we can observe its inputs and outputs, but we cannot tell how one becomes the other. We face these two meanings daily: tracked ever more closely by firms and government, we have no clear idea of just how far much of this information can travel, how it is used, or its consequences.⁷

The Power of Secrecy

Knowledge is power. To scrutinize others while avoiding scrutiny oneself is one of the most important forms of power.⁸ Firms seek out intimate details of potential customers’ and employees’ lives, but give regulators as little information as they possibly can about their own statistics and procedures.⁹ Internet companies collect more and more data on their users but fight regulations that would let those same users exercise some control over the resulting digital dossiers.

As technology advances, market pressures raise the stakes of the data game. Surveillance cameras become cheaper every year; sensors are embedded in more places.¹⁰ Cell phones track our movements; programs log our keystrokes. New hardware and new software promise to make “quantified selves” of all of us, whether we like it or not.¹¹ The resulting information—a vast amount of data that until recently went unrecorded—is fed into databases and assembled into profiles of unprecedented depth and specificity.

But to what ends, and to whose? The decline in personal privacy might be worthwhile if it were matched by comparable levels of transparency from corporations and government. But for the most part it is not. Credit raters, search engines, major banks, and the TSA take in data about us and convert it into scores, rankings, risk calculations, and watch lists with vitally important consequences. But the proprietary algorithms by which they do so are immune from scrutiny, except on the rare occasions when a whistleblower litigates or leaks.

Sometimes secrecy is warranted. We don’t want terrorists to be able to evade detection because they know exactly what Homeland Security agents are looking out for.¹² But when every move we make is subject to inspection by entities whose procedures and personnel are exempt from even remotely similar treatment, the promise of democracy and free markets rings hollow. Secrecy is approaching critical mass, and we are in the dark about crucial decisions. Greater openness is imperative.

Reputation, Search, Finance

At the core of the information economy are Internet and finance companies that accumulate vast amounts of digital data, and with it intimate details of their customers’—our—lives. They use it to make important decisions about us and to influence the decisions we make for ourselves. But what do we know about them? A bad credit score may cost a borrower hundreds of thousands of dollars, but he will never understand exactly how it was calculated. A predictive analytics firm may score someone as a “high cost” or “unreliable” worker, yet never tell her about the decision.

More benignly, perhaps, these companies influence the choices we make ourselves. Recommendation engines at Amazon and YouTube affect an automated familiarity, gently suggesting offerings they think we’ll like. But don’t discount the significance of that “perhaps.” The economic, political, and cultural agendas behind their suggestions are hard to unravel. As middlemen, they specialize in shifting alliances, sometimes advancing the interests of customers, sometimes suppliers: all to orchestrate an online world that maximizes their own profits.

Financial institutions exert direct power over us, deciding the terms of credit and debt. Yet they too shroud key deals in impenetrable layers of complexity. In 2008, when secret goings-on in the money world provoked a crisis of trust that brought the banking system to the brink of collapse, the Federal Reserve intervened to stabilize things—and kept key terms of those interventions secret as well. Journalists didn’t uncover the massive scope of its interventions until late 2011.¹³ That was well after landmark financial reform legislation had been debated and passed—without informed input from the electorate—and then watered down by the same corporate titans whom the Fed had just had to bail out.

Reputation. Search. Finance. These are the areas in which Big Data looms largest in our lives. But too often it looms invisibly, undermining the openness of our society and the fairness of our markets. Consider just a few of the issues raised by the new technologies of ranking and evaluation:

1. Should a credit card company be entitled to raise a couple’s interest rate if they seek marriage counseling? If so, should cardholders know this?
2. Should Google, Apple, Twitter, or Facebook be able to shut out websites or books entirely, even when their content is completely legal? And if they do, should they tell us?
3. Should the Federal Reserve be allowed to print unknown sums of money to save banks from their own scandalous behavior? If so, how and when should citizens get to learn what’s going on?
4. Should the hundreds of thousands of American citizens placed on secret “watch lists” be so informed, and should they be given the chance to clear their names?

The leading firms of Wall Street and Silicon Valley are not alone in the secretiveness of their operations, but I will be focusing primarily on them because of their unique roles in society. While accounting for “less than 10% of the value added” in the U.S. economy in the fourth quarter of 2010, the finance sector took 29 percent—$57.7 billion—of profits.¹⁴ Silicon Valley firms are also remarkably profitable, and powerful.¹⁵ What finance firms do with money, leading Internet companies do with attention. They direct it toward some ideas, goods, and services, and away from others. They organize the world for us, and we have been quick to welcome this data-driven convenience. But we need to be honest about its costs.

Secrecy and Complexity

Deconstructing the black boxes of Big Data isn’t easy. Even if they were willing to expose their methods to the public, the modern Internet and banking sectors pose tough challenges to our understanding of those methods. The conclusions they come to—about the productivity of employees, or the relevance of websites, or the attractiveness of investments—are determined by complex formulas devised by legions of engineers and guarded by a phalanx of lawyers.

In this book, we will be exploring three critical strategies for keeping black boxes closed: “real” secrecy, legal secrecy, and obfuscation. Real secrecy establishes a barrier between hidden content and unauthorized access to it. We use real secrecy daily when we lock our doors or protect our e-mail with passwords. Legal secrecy obliges those privy to certain information to keep it secret; a bank employee is obliged both by statutory authority and b...

Table of contents

• Cover
• Title Page
• Copyright
• Dedication
• Contents
• Epigraphs
• 1. Introduction—The Need to Know
• 2. Digital Reputation in an Era of Runaway Data
• 3. The Hidden Logics of Search
• 4. Finance’s Algorithms: The Emperor’s New Codes
• 5. Watching (and Improving) the Watchers
• 6. Toward an Intelligible Society
• Notes
• Acknowledgments
• Index