1
INTRODUCTIONâTHE NEED TO KNOW
EVERYBODY KNOWS the story about the man crawling intently around a lamppost on a dark night. When a police officer comes along and wants to know what heâs doing, he says heâs looking for his keys. âYou lost them here?â asks the cop. âNo,â the seeker replies, âbut this is where the light is.â This bromide about futility has lately taken on a whole new meaning as a metaphor for our increasingly enigmatic technologies.
Thereâs a noble tradition among social scientists of trying to clarify how power works: who gets what, when, where, and why.1 Our common life is explored in books like The Achieving Society, The Winner-Take-All Society, The Good Society, and The Decent Society. At their best, these works also tell us why such inquiry matters.2
But efforts like these are only as good as the information available. We cannot understand, or even investigate, a subject about which nothing is known. Amateur epistemologists have many names for this problem. âUnknown unknowns,â âblack swans,â and âdeep secretsâ are popular catchphrases for our many areas of social blankness.3 There is even an emerging field of âagnotologyâ that studies the âstructural production of ignorance, its diverse causes and conformations, whether brought about by neglect, forgetfulness, myopia, extinction, secrecy, or suppression.â4
Gaps in knowledge, putative and real, have powerful implications, as do the uses that are made of them. Alan Greenspan, once the most powerful central banker in the world, claimed that todayâs markets are driven by an âunredeemably opaqueâ version of Adam Smithâs âinvisible hand,â and that no one (including regulators) can ever get âmore than a glimpse at the internal workings of the simplest of modern financial systems.â If this is true, libertarian policy would seem to be the only reasonable response. Friedrich von Hayek, a preeminent theorist of laissez-faire, called the âknowledge problemâ an insuperable barrier to benevolent government interventions in the economy.5
But what if the âknowledge problemâ is not an intrinsic aspect of the market, but rather is deliberately encouraged by certain businesses? What if financiers keep their doings opaque on purpose, precisely to avoid or to confound regulation? That would imply something very different about the merits of deregulation.
The challenge of the âknowledge problemâ is just one example of a general truth: What we do and donât know about the social (as opposed to the natural) world is not inherent in its nature, but is itself a function of social constructs. Much of what we can find out about companies, governments, or even one another, is governed by law. Laws of privacy, trade secrecy, the so-called Freedom of Information Actâall set limits to inquiry. They rule certain investigations out of the question before they can even begin. We need to ask: To whose benefit?
Some of these laws are crucial to a decent society. No one wants to live in a world where the boss can tape our bathroom breaks. But the laws of information protect much more than personal privacy. They allow pharmaceutical firms to hide the dangers of a new drug behind veils of trade secrecy and banks to obscure tax liabilities behind shell corporations. And they are much too valuable to their beneficiaries to be relinquished readily.
Even our political and legal systems, the spaces of our common life that are supposed to be the most open and transparent, are being colonized by the logic of secrecy. The executive branch has been lobbying ever more forcefully for the right to enact and enforce âsecret lawâ in its pursuit of the âwar on terror,â and voters contend in an electoral arena flooded with âdark moneyââdollars whose donors, and whose influence, will be disclosed only after the election, if at all.6
But while powerful businesses, financial institutions, and government agencies hide their actions behind nondisclosure agreements, âproprietary methods,â and gag rules, our own lives are increasingly open books. Everything we do online is recorded; the only questions left are to whom the data will be available, and for how long. Anonymizing software may shield us for a little while, but who knows whether trying to hide isnât itself the ultimate red flag for watchful authorities? Surveillance cameras, data brokers, sensor networks, and âsupercookiesâ record how fast we drive, what pills we take, what books we read, what websites we visit. The law, so aggressively protective of secrecy in the world of commerce, is increasingly silent when it comes to the privacy of persons.
That incongruity is the focus of this book. How has secrecy become so important to industries ranging from Wall Street to Silicon Valley? What are the social implications of the invisible practices that hide the way people and businesses are labeled and treated? How can the law be used to enact the best possible balance between privacy and openness? To answer these questions is to chart a path toward a more intelligible social order.
But first, we must fully understand the problem. The term âblack boxâ is a useful metaphor for doing so, given its own dual meaning. It can refer to a recording device, like the data-monitoring systems in planes, trains, and cars. Or it can mean a system whose workings are mysterious; we can observe its inputs and outputs, but we cannot tell how one becomes the other. We face these two meanings daily: tracked ever more closely by firms and government, we have no clear idea of just how far much of this information can travel, how it is used, or its consequences.7
The Power of Secrecy
Knowledge is power. To scrutinize others while avoiding scrutiny oneself is one of the most important forms of power.8 Firms seek out intimate details of potential customersâ and employeesâ lives, but give regulators as little information as they possibly can about their own statistics and procedures.9 Internet companies collect more and more data on their users but fight regulations that would let those same users exercise some control over the resulting digital dossiers.
As technology advances, market pressures raise the stakes of the data game. Surveillance cameras become cheaper every year; sensors are embedded in more places.10 Cell phones track our movements; programs log our keystrokes. New hardware and new software promise to make âquantified selvesâ of all of us, whether we like it or not.11 The resulting informationâa vast amount of data that until recently went unrecordedâis fed into databases and assembled into profiles of unprecedented depth and specificity.
But to what ends, and to whose? The decline in personal privacy might be worthwhile if it were matched by comparable levels of transparency from corporations and government. But for the most part it is not. Credit raters, search engines, major banks, and the TSA take in data about us and convert it into scores, rankings, risk calculations, and watch lists with vitally important consequences. But the proprietary algorithms by which they do so are immune from scrutiny, except on the rare occasions when a whistleblower litigates or leaks.
Sometimes secrecy is warranted. We donât want terrorists to be able to evade detection because they know exactly what Homeland Security agents are looking out for.12 But when every move we make is subject to inspection by entities whose procedures and personnel are exempt from even remotely similar treatment, the promise of democracy and free markets rings hollow. Secrecy is approaching critical mass, and we are in the dark about crucial decisions. Greater openness is imperative.
Reputation, Search, Finance
At the core of the information economy are Internet and finance companies that accumulate vast amounts of digital data, and with it intimate details of their customersââourâlives. They use it to make important decisions about us and to influence the decisions we make for ourselves. But what do we know about them? A bad credit score may cost a borrower hundreds of thousands of dollars, but he will never understand exactly how it was calculated. A predictive analytics firm may score someone as a âhigh costâ or âunreliableâ worker, yet never tell her about the decision.
More benignly, perhaps, these companies influence the choices we make ourselves. Recommendation engines at Amazon and YouTube affect an automated familiarity, gently suggesting offerings they think weâll like. But donât discount the significance of that âperhaps.â The economic, political, and cultural agendas behind their suggestions are hard to unravel. As middlemen, they specialize in shifting alliances, sometimes advancing the interests of customers, sometimes suppliers: all to orchestrate an online world that maximizes their own profits.
Financial institutions exert direct power over us, deciding the terms of credit and debt. Yet they too shroud key deals in impenetrable layers of complexity. In 2008, when secret goings-on in the money world provoked a crisis of trust that brought the banking system to the brink of collapse, the Federal Reserve intervened to stabilize thingsâand kept key terms of those interventions secret as well. Journalists didnât uncover the massive scope of its interventions until late 2011.13 That was well after landmark financial reform legislation had been debated and passedâwithout informed input from the electorateâand then watered down by the same corporate titans whom the Fed had just had to bail out.
Reputation. Search. Finance. These are the areas in which Big Data looms largest in our lives. But too often it looms invisibly, undermining the openness of our society and the fairness of our markets. Consider just a few of the issues raised by the new technologies of ranking and evaluation:
- Should a credit card company be entitled to raise a coupleâs interest rate if they seek marriage counseling? If so, should cardholders know this?
- Should Google, Apple, Twitter, or Facebook be able to shut out websites or books entirely, even when their content is completely legal? And if they do, should they tell us?
- Should the Federal Reserve be allowed to print unknown sums of money to save banks from their own scandalous behavior? If so, how and when should citizens get to learn whatâs going on?
- Should the hundreds of thousands of American citizens placed on secret âwatch listsâ be so informed, and should they be given the chance to clear their names?
The leading firms of Wall Street and Silicon Valley are not alone in the secretiveness of their operations, but I will be focusing primarily on them because of their unique roles in society. While accounting for âless than 10% of the value addedâ in the U.S. economy in the fourth quarter of 2010, the finance sector took 29 percentâ$57.7 billionâof profits.14 Silicon Valley firms are also remarkably profitable, and powerful.15 What finance firms do with money, leading Internet companies do with attention. They direct it toward some ideas, goods, and services, and away from others. They organize the world for us, and we have been quick to welcome this data-driven convenience. But we need to be honest about its costs.
Secrecy and Complexity
Deconstructing the black boxes of Big Data isnât easy. Even if they were willing to expose their methods to the public, the modern Internet and banking sectors pose tough challenges to our understanding of those methods. The conclusions they come toâabout the productivity of employees, or the relevance of websites, or the attractiveness of investmentsâare determined by complex formulas devised by legions of engineers and guarded by a phalanx of lawyers.
In this book, we will be exploring three critical strategies for keeping black boxes closed: ârealâ secrecy, legal secrecy, and obfuscation. Real secrecy establishes a barrier between hidden content and unauthorized access to it. We use real secrecy daily when we lock our doors or protect our e-mail with passwords. Legal secrecy obliges those privy to certain information to keep it secret; a bank employee is obliged both by statutory authority and b...