Open Source Intelligence in the Twenty-First Century

New Approaches and Opportunities

C. Hobbs, M. Moran, D. Salisbury, C. Hobbs, M. Moran, D. Salisbury

• Publisher: Palgrave Macmillan
• Year: 2014
• ISBN:
  9781137353320
• Topic: Politik & Internationale Beziehungen
• Subtopic: Frieden & globale Entwicklung

This edited book provides an insight into the new approaches, challenges and opportunities that characterise open source intelligence (OSINT) at the beginning of the twenty-first century. It does so by considering the impacts of OSINT on three important contemporary security issues: nuclear proliferation, humanitarian crises and terrorism.

Part I

Open Source Intelligence: New Methods and Approaches

1

Exploring the Role and Value of Open Source Intelligence

Stevyn D. Gibson

A proper analysis of the intelligence obtainable by these overt, normal and aboveboard means would supply us with over 80 percent, I should estimate, of the information required for the guidance of our national policy.

Allen Dulles

OSINT and the mythology of the ‘80 percent rule’

Allen Dulles’s testimony to the Senate Committee on Armed Services on 25 April 1947 was only nine pages long and was hastily written¹ but, in it, he began the process of the demystification of the art of intelligence, adding:

Because of its glamour and mystery, overemphasis is generally placed on what is called secret intelligence, namely the intelligence that is obtained by secret means and by secret agents … In time of peace the bulk of intelligence can be obtained through overt channels, through our diplomatic and consular missions, and our military, naval and air attachés in the normal and proper course of their work. It can also be obtained through the world press, the radio, and through the many thousands of Americans, business and professional men and American residents of foreign countries, who are naturally and normally brought in touch with what is going on in those countries.

Dulles’s 80 per cent claim represented the first notable attempt to quantify the open source contribution to intelligence and has since been widely repeated by practitioners, commentators and customers alike. The North Atlantic Treaty Organisation (NATO) considers that the exploitation of OSINT provides 80 per cent of the final product for arms control and arms proliferation issues.² For his part, Hulnick suggests that 80 per cent of US Cold War analysis could have been taken from open sources.³ Steele’s view is that, across the board, OSINT can provide 80 per cent of what any government needs to know and 90 per cent for private sector organisations.⁴ For some collection agencies, 90 per cent is the norm.⁵ The Central Intelligence Agency’s (CIA’s) Bin Laden unit, for example, noted that ‘90 percent of what you need to know comes from open source intelligence’.⁶ EUROPOL goes further, suggesting that the contribution might be as high as 95 per cent for counterterrorism issues.⁷ Similarly, the 1996 Aspin-Brown Commission remarked that ‘In some areas … it is estimated that as much as 95 per cent of the information utilised now comes from open sources’.⁸ Responding to the report of the 1997 US Commission on Secrecy, the grand-master of US foreign policy, George F. Kennan, wrote to US Commission Chairman Senator Daniel P. Moynihan⁹:

It is my conviction, based on some 70 years of experience, first as a Government official and then in the past 45 years as a historian, that the need by our government for secret intelligence about affairs elsewhere in the world has been vastly overrated. I would say that something upward of 95 per cent of what we need to know could be very well obtained by the careful and competent study of perfectly legitimate sources of information open and available to us in the rich library and archival holdings of this country. Much of the remainder, if it could not be found here (and there is very little of it that could not), could easily be nonsecretively elicited from similar sources abroad.

Finally, but not exhaustively, in December 2005, former Deputy Assistant Director of Central Intelligence W.M. Nolte stated that 95–98 per cent of all information handled by the US intelligence community derives from open sources.¹⁰ It would seem, then, that those best placed to judge regard OSINT as constituting a significant majority of the intelligence effort.

Regardless of the general attribution and subjective estimate of OSINT’s efficiency, the obvious question ought to be: what is it 80 per cent or 95 per cent of? Is it output measured by paragraphs in a final intelligence report, actions enumerated by arrests and threat interdictions, or clearly observable policy achievements?

Thus, this oft-quoted estimate that 80 per cent or more of final intelligence product is generated from open source exploitation is a mischievous ‘red herring’. How is the figure calculated? What is the yardstick of measurement? Where are the repeatable, corroborable data by which it is determined? The evidence is anecdotal; subjectively assessed rather than methodically derived. Furthermore, while Dulles and Kennan suggest that 80 per cent of what the intelligence function needs to know can be found from open source information, this is not quite the same as saying that 80 per cent of what the intelligence function ‘knows’ is OSINT. The former is a claim for the usefulness of OSINT as a deliberate discipline in itself; the latter is a verdict on the limitations of traditional secret sources.

Moreover, this broad 80 per cent estimate does not equate to all intelligence subjects equally, or simultaneously. In the mid-1990s the US Government’s Community Open Source Program (COSP) estimated the OSINT contribution to be in the range of 40 per cent overall, while specific contributions, depending upon target difficulty, ranged from 10 per cent in denied-area, secret-issue matters to 90 per cent in international economics. This COSP estimate may be the only methodically derived data point for the evaluation of OSINT’s contribution. As Markowitz suggests, much of the chatter surrounding the 80 per cent claim might be no more than circular reporting of Dulles’s original estimate. Once stated by respected members of the intelligence community it passes into lore.

Therefore these ‘estimates’ of OSINT’s contribution to final intelligence product, even if correct, are merely expressions of efficiency – inputs related to outputs. This may form the basis of an important argument about the allocation of scarce intelligence resources. Although, in terms of those scarce resources devoted to OSINT, this 80 per cent label neither carries the weight of an 800 lb gorilla nor enhances its ‘second-class’ status relative to closed INTs. Most importantly it does not reveal any understanding of OSINT’s contribution to decision- or policy-making effectiveness.

In this context, it would seem more useful to explore the effectiveness of OSINT – what it can do for the intelligence function. In this regard, it can replicate ‘secret’ sources, form the matrix to bind all other intelligence sources together and still has its own distinct attributes to offer. Yet OSINT is no more a ‘silver bullet’ for policy than closed sources.

Some ‘INTs’ are more equal than others

Regardless of the ‘80 percent rule’, the proportion of resources devoted to OSINT is nowhere near comparable to closed. OSINT may be formally and deliberately exploited within the intelligence community, but it is regarded as less equal than others. This is due to a number of cultural barriers that have somehow come to shape OSINT’s potential.

Many myths and misconceptions serve to confound OSINT’s contribution to intelligence: that it is in competition with secret intelligence, rather than complementary to, if not thoroughly enmeshed with, closed; that it resides solely on the Internet rather than in magnetic, film, paper and other non-digital sources; that it is exclusively text-based and in English rather than also oral, image-based, and in many languages; that it is conducted overtly, when collectors may hide their interest at a conference, mask their intentions in the academic papers that they deliver or ‘anonymise’ their IP address when interrogating websites; that it is exclusive to the public sector, when, by definition, it is available to many with a cause, including the private, academic and other non-governmental sectors; that it is free to collect or assess, rather than requiring specialised effort and increasingly expensive effort as greater value is added; that the greatest added value may come from any sector – private sector product is not ‘inferior’ to public sector product, nor OSINT necessarily ‘inferior’ to closed ‘INTs’; that it is excused the usual ‘rules’ of information-working commonly applied to construct assessment in support of decision-making, rather than be validated for accuracy, relevance and timeliness in the same way that journalism and research should be; and, not least, that OSINT cannot provide a ‘smoking gun’, when many historical examinations of ‘intelligence surprise’ show those surprises being pre-trailed in the press, and countless examples of contemporary social networking media ‘confessions’ demonstrate that much evidence is already in plain sight in these media.¹¹

Thus the concern with effectiveness is important in one key regard – it should prioritise, or at least influence, the treatment of OSINT within agencies and across national intelligence machineries. All of these cultural, organisational and technical misconceptions underline the necessity for a distinct OSINT tradecraft, appropriate tools and techniques, specialised software and equipment, a ‘familiarity’ with contemporary information and communication technology (ICT) and a befitting budget.¹² The establishment of the Open Source Center (OSC) in the US goes part way towards realising such a vision; but, as Bean observes, the OSC remains inside a closed environment and subject to high-level office politics.¹³

Any obsession should be with the meaning rather than the number. As RAND and Gill and Phythian have all noted, intelligence effectiveness is a slippery concept to pin down, let alone measure.¹⁴ Odom similarly bemoans the fact that nowhere within the intelligence community are inputs related to outputs:

Because the DCI has never made the effort to impose a similar system (to the Defense Department) on resource management in the Intelligence Community, its consolidated Intelligence Community budget does not effectively relate inputs to outputs.¹⁵

Odom argues that instead of traditional benchmarks of quantity and quality of data gathered, effectiveness should be measured by how much output is used by, and meets the needs of, its customers. Interestingly, the US Army begins to construct such an argument: ‘determining whether PIRs [priority intelligence requirements] have been answered’.¹⁶ The US Joint Chiefs expand it by recognising that intelligence evaluation is undertaken by the customer, based upon ‘the attributes of good intelligence: anticipatory, timely, accurate, usable, complete, relevant, objective, available’.¹⁷

The UK intelligence community similarly confuses efficiency with effectiveness. Its Intelligence and Security Committee relates inputs to outputs through ‘top-level management tools’ ensuring that ‘business’ objectives are met within the intelligence agencies, the resulting public service agreements and service delivery agreements reflecting a transfer to resource–based accounting processes. Worse, they equate process with purpose and confuse means with ends.

Thus today’s ‘best’ determination of effectiveness seems to reside with the customer or the accountant. Yet customer satisfaction, business targets and balancing scorecards generate little meaningful insight into the effectiveness of intelligence in relation to policy objectives. In order to evaluate OSINT’s contribution, it seems crucial to understand how it is effective, both absolutely within the intelligence function and relative to closed intelligence.

Ecclesiastes Chapter 1, Verse 9

The thing that hath been, it is that which shall be;

and that which is done is that which shall be done:

and there is no new thing under the sun.

Standard King James Bible

In 1808, Wellington assembled his generals before departing for the Peninsular War and admonished them for their ignorance of Napoleon’s new French infantry formations being openly reported in The Times. In 1826, Henry Brougham, the radical Whig politician, established the Society for the Diffusion of Useful Knowledge. Its aim was ‘to impart useful information to all classes of the community’.¹⁸ This utopian dream of knowledge transforming society sounds strikingly similar to the present-day goal of Google.¹⁹ The society closed in 1848. Its product was considered erratic and miscellaneous; one might add idealist, naïve and absent of political purpose. It thought that the provision of information was the end game and in doing so elevated means to ends. Contemporary open source evangelists, such as those behind the now infamous WikiLeaks, pioneer similar utopian visions for ‘open’ information. Finally, during the nineteenth century, the commercial open source publication Jane’s Fighting Ships was established in 1898.

In March 2002, John Darwin canoed out into the North Sea from the English seaside town of Seaton Carew and faked his suicide. By February 2003 he had moved ...