Empire of Secrets

British Intelligence, the Cold War and the Twilight of Empire

Calder Walton

• Publisher: HarperPress
• Year: 2013
• Print ISBN:
  9780007457977
• eBook ISBN:
  9780007468423
• Topic: History
• Subtopic: 20th Century History
• Index: History

The winner of the 2013 Longman-History Today Book Prize is the gripping and largely untold story of the role of the intelligence services in Britain's retreat from empire.

Against the background of the Cold War, and the looming spectre of Soviet-sponsored subversion in Britain's dwindling colonial possessions, the imperial intelligence service MI5 played a crucial but top secret role in passing power to newly independent national states across the globe.

Mining recently declassified intelligence records, Calder Walton reveals this 'missing link' in Britain's post-war history. He sheds new light on everything from violent counter-insurgencies fought by British forces in the jungles of Malaya and Kenya, to urban warfare campaigns conducted in Palestine and the Arabian Peninsula. Drawing on a wealth of previously classified documents, as well as hitherto overlooked personal papers, this is also the first book to draw on records from the Foreign Office's secret archive at Hanslope Park, which contains some of the darkest and most shameful secrets from the last days of Britain's empire.

Packed with incidents straight out of a John le Carré novel, Empire of Secrets is an exhilarating read by an exciting new voice in intelligence history.

1

Victoria’s Secrets: British Intelligence and Empire Before the Second World War

One advantage of the secret service is that it has no worrying audit. The service is ludicrously starved, of course, but the funds are administered by a few men who do not call for vouchers or present itemised accounts …

He considered the years to come when Kim would have been entered and made to the Great Game that never ceases day and night, throughout India.

RUDYARD KIPLING, Kim¹

Governments have conducted espionage and intelligence-gathering efforts for centuries. Indeed, intelligence-gathering – often said to be the world’s second oldest profession – is as old as governments themselves. In Britain, there was a ‘secret service’ operating at least since the reign of Queen Elizabeth I in the sixteenth century, which under Sir Francis Walsingham was tasked to gather intelligence on the Spanish Armada and to uncover various Catholic intrigues and plots. However, it was not until the nineteenth century, and more importantly the early twentieth century, that the British government began to devote significant resources to intelligence, and turn it into a professional, bureaucratic enterprise. Despite Britain’s long history of clandestine espionage work, in fact it was not in the ‘domestic’ realm that its intelligence-gathering was to develop most rapidly. Instead, it was in the British empire, which in the course of the nineteenth and twentieth centuries grew to become the greatest empire in world history, that intelligence found a particularly important role.²

From the earliest days of the British intelligence community, which was established in the early twentieth century, there was a close connection between intelligence-gathering and empire. It is not an exaggeration to say that in its early years British intelligence was British imperial intelligence.This is not surprising when it is considered that intelligence played an essential role in the administration of the empire, which by the 1920s had grown to encompass one-quarter of the world’s territory and population. After 1918, as one geographer proudly commented, the empire reached its widest extent, covering ‘one continent, a hundred peninsulas, five hundred promontories, a thousand lakes, two thousand rivers, ten thousand islands’. The empire had four kinds of dependent territories: colonies, protectorates, protected states and trust territories. At one end of the spectrum, colonies were those territories, like Kenya, where the monarch of the United Kingdom had absolute sovereignty, while trust territories, at the other end of the spectrum, were those assigned to Great Britain for administration under a special mandate, like Palestine. There was often little practical difference between colonies and protectorates. The Colonial Office usually referred to territories under ‘traditional’ rulers, with a British resident, as ‘protected states’. The typology of these dependent states was incredibly confusing (sometimes even to the Colonial Office itself).

One reason for the importance of intelligence in the empire was the lack of sheer manpower required to cover such enormous territories. Even at its height, British rule in India was maintained through an incredibly small number of administrative officials, with the renowned Indian Civil Service in the Raj boasting a total of just 1,200 posts, at a time when the population of India was probably around 280 million. Before 1939 the Indian army of 200,000 men, together with a British garrison of 60,000, was responsible for keeping the peace on land from Egypt to Hong Kong – British territories ‘East of Suez’, to use the phrase from the time. With such meagre resources at its disposal, British rule in India required up-to-date and reliable information on its enemies, both imagined and real. This was acquired through networks of informants and agents, and from intercepted communications. It is little wonder that, as one study has termed it, the British empire in the nineteenth century was an ‘empire of information’.³

Intelligence-gathering also came to the forefront in Britain’s imperial military campaigns in the late nineteenth and early twentieth centuries. One of the most exhilarating theatres for intelligence operations, or spying, lay in India’s North-West Frontier – now the tribal borderlands of Pakistan – where Victorian Britain fought the ‘Great Game’ with Russia, a conflict memorably portrayed by Rudyard Kipling in Kim, arguably one of the greatest espionage novels of all time. In Kim, Kipling described the ‘Great Game’ as essentially an intelligence conflict, which ‘never ceases day or night’, with both Britain and Russia running spies and informants to discover the other’s intentions. However, the reality was that it was often not difficult for Russia to spot British imperial intelligence agents: they were often extremely amateurish and deployed flimsy covers, variously posing as butterfly collectors, archaeologists and ethnographers. That said, it was in the ‘Great Game’ that some distinctly more professional forms of intelligence-gathering were born, particularly in a process that would later become known as signals intelligence (SIGINT), the interception and decryption of communications, or ‘signals’. In 1844 the Indian army pioneered one of the first permanent code-breaking bureaus in the world, which gained notable successes in reading Russian communications long before any similar European SIGINT agency had done so. The British military also made innovative use of intelligence during its campaigns in Egypt in the 1880s, successfully deploying a series of agents and scouts to reconnoitre the location of Egyptian forces in the desert.⁴

The very process of Britain’s colonial expansion in the Victorian period, especially during the so-called ‘scramble for Africa’ beginning in the 1880s, necessitated new forms of systematic intelligence-gathering, such as mapping and census-taking. In undertaking such activities, Britain was not acting differently from its imperial rivals at the time – France, Russia, Germany and Italy. Before any colonial power could dominate, control and exploit colonial populations, in Africa or elsewhere, it first had to map them. In practice, however, the process of mapping an empire often ignored its realities. Maps imposed European geometrical patterns on amorphous landscapes, drawing frontiers that cut through tribal communities as well as ethnic and linguistic groups. To this day, it is not difficult to spot the borders of those countries, particularly in Africa, which were drawn by European cartographers: many are arranged at right angles and slice through geographical features and ethnographic groupings. Sometimes European powers displaced and resettled colonial populations in order to make them reflect the ethnographic colonial maps. In the ‘white man’s burden’ of colonial rule, subtle realities did not matter.⁵

Given all that, it is no coincidence that Britain’s first Directorate of Military Intelligence, established in 1887, grew from the Topographical and Statistical Department in the War Office, which was responsible for mapping much of the British empire. Moreover, it was a violent colonial ‘small war’ in an outpost of the British empire, the Second Anglo-Boer War in southern Africa, waged between 1899 and 1902, which first alerted the British government to the need for establishing a permanent intelligence service. The so-called Boer War exposed to Britain’s military leaders, the Chiefs of Staff in London, how fragile the nation’s colonial holdings were. It took the British military much longer than expected, three years, and also the deployment of some 45,000 troops, to defeat a group of rebellious Dutch Boer farmers in the Cape Colony (now South Africa) who harried the British Army through guerrilla warfare. In fighting the insurgency there, it has to be noted that the British military developed some ominous strategies, not least the establishment of ‘concentration camps’, or detention camps, where suspected insurgents were ‘concentrated’. This type of warfare, in which the distinction between combatants and non-combatant civilians was blurred, was to have horrific echoes in the twentieth century. As far as intelligence was concerned, the kind of irregular warfare that Britain faced in the Boer War, like that experienced by other European powers in their own colonial ‘small wars’ – literally guerrilla in Spanish – revealed the paramount need for effective intelligence-gathering. In fact, it was during the Boer War that a British officer, Lt. Col. David Henderson, wrote an influential paper for the War Office in London, ‘Field Intelligence: its principles and practice’, which became the basis of a manual, ‘Regulations for intelligence duties in the field’, published by the War Office in 1904. This manual became the inspiration for the British Army’s intelligence corps, founded ten years later, on the outbreak of the First World War.⁶

Despite Britain’s long history of intelligence-gathering, a watershed occurred in the early twentieth century. Partly in response to fears of Britain’s colonial frailty, as revealed by the Boer War, but more specifically as a result of fears about the growing threat posed by the German empire, in October 1909 the British government took the momentous decision to establish a permanent, peacetime intelligence department. This decision was taken by the Committee of Imperial Defence – significantly, it was imperial defence that led to the setting up of Britain’s spook agencies. The department, known as the ‘Secret Service Bureau’, was divided into two branches. The ‘domestic’ branch, MO5(g), was responsible for security intelligence – counter-espionage, counter-sabotage and counter-subversion. During the First World War MO5(g) was renamed Military Intelligence 5, or ‘MI5’, and after the war it was again rechristened the Security Service – twin designations (the Security Service, MI5) that it keeps to the present day. Sir Vernon Kell, a retired officer from the South Staffordshire Regiment, served as Director-General of MI5 from 1909 to 1940, roughly one-third of its history to date, making him the longest-ever serving head of any British government department.⁷

Meanwhile, the ‘foreign’ branch of the Secret Service Bureau, first known as MI1C, was renamed Military Intelligence 6, or ‘MI6’, during the First World War. Thereafter it became known as MI6 or the Secret Intelligence Service (SIS) – again, twin designations that it retains to the present day. Its first head was Sir Mansfield Cumming, a Royal Navy officer who had taken early retirement due to ill-health. By all accounts he was a remarkable character. In the early stages of the First World War he lost a leg in a road traffic accident in France – as the story goes, he hacked his own leg off with a pocket penknife in order to drag himself to safety from the wreckage of his car. This accident caused him to use a wheelchair, and colleagues later recalled that he would terrorise the corridors of power in Whitehall, spinning at high speeds around corners.⁸

In taking the decision to establish a professional intelligence department in 1909, the British government actually came late to the ‘intelligence game’ when compared to other European powers, most of which had already set up such bodies by the turn of the twentieth century. France had established code-breaking ‘black chambers’ (cabinets noirs) in the middle of the nineteenth century, while tsarist Russia had an infamous intelligence service (the Okhrana), and Germany had a specialised intelligence service (Nachrichtendienst) operating at least since the time of the Franco-Prussian war in 1870. The reason for Britain’s late arrival into the world of espionage was due to strong opposition from some Victorian and Edwardian politicians, who decried ‘intelligence’ as an inherently un-English pursuit: gentlemen ‘did not read each other’s mail’, went the phrase, and ‘espionage’ was not even an English word, as some liked to point out. It was better to leave such sordid exploits to the Continental powers, where they belonged.⁹

The formation of the two services that would later become known as MI5 and SIS represented a fundamental break with all British intelligence-gathering efforts up to that point. For the first time, the government had professional, dedicated peacetime intelligence services at its disposal. Operational distinctions between MI5 and SIS, particularly jurisdictional disputes over what constituted ‘domestic’ and ‘foreign’ territory, proved a thorny subject that would only be resolved over subsequent decades. Nevertheless, the crucial point is that, unlike all British intelligence-gathering efforts up to that point, after 1909 the government was equipped with independent intelligence bureaucracies, furnished with card-catalogue index registries, which brought together information from all available sources. Whereas previously the British military and various government departments, such as the India Office, had gathered intelligence and conducted espionage for their own purposes, often on an ad hoc basis, the services established in 1909 had two specific combined purposes: to gather and assess intelligence. They were also inter-departmental, that is to say they were meant to ‘service’ all British government departments with the intelligence they needed. Although MI5 and SIS grew out of Britain’s military intelligence department (MO5), they were different from the intelligence departments of the armed forces, which were not inter-departmental. All three of the armed services, the army, Royal Navy and Royal Air Force, would go on to maintain their own intelligence departments, but it is MI5 and SIS (and later GC&CS) that are usually understood to be Britain’s intelligence services, or, more amorphously, ‘British intelligence’. The establishment of MI5 and SIS also witnessed for the first time a distinction between various grades of classified information (or ‘intelligence’), such as ‘secret’ and ‘top secret’. Thus, while British government departments before 1909 had gathered intelligence, and would continue to do so thereafter, the breakthrough for the government was that after 1909 it had for the first time its own intelligence services.

To this day, MI5 and SIS retain many of the practices established in their earliest days. The Chiefs of SIS retain the designation ‘C’, a title that was first used by Sir Mansfield Cumming, which is variously understood to stand either for ‘Cumming’ or for ‘Chief’. Other SIS rituals established in its earliest times which continue to the present include a green light outside C’s room (indicating that C is busy), special green ink that is reserved for him alone to use, and the ubiquitous and sometimes pointless use of codenames. SIS reports are still referred to as ‘CX reports’, apparently meaning ‘C Exclusively’. Similar continuities also exist in MI5. The terms ‘Put Away’ (‘P/A’) and ‘Look Up’ (‘L/U’), for example, can be seen on the front of countless declassified MI5 records, indicating when a file has been looked up and then put away in a secure cabinet – both of which were terms used by Kell soon after his ‘Bureau’ was established. The same is true of ‘Nothing Recorded Against’ (‘NRA’), which refers to one of the most important, but least glamorous, activities that MI5 officers have undertaken since Kell’s time: when an MI5 officer has looked up an individual in the service’s central archive, but has found nothing incriminating.

Eccentric rituals and designations apart, MI5 and SIS also retain much more important legacies from their early history. From the outset of their operations it was established that neither would have any executive powers. In contrast to law-enforcement agencies such as London’s Special Branch at Scotland Yard, or the US Federal Bureau of Investigation (FBI), neither MI5 nor SIS has ever had any powers of arrest. Readers may be disappointed to learn that SIS officers have never had ‘licences to kill’. Instead, MI5 and SIS have always relied on police authorities in Britain, particularly Special Branch, to carry out arrests for them. This was a calculated strategy on the part of the Chiefs of Staff and the British government. By decoupling intelligence-gathering from law enforcement, policing, and executive action more generally, they hoped to avoid the establishment of a ‘police state’, which they feared would be created by providing the secret services with powers of arrest. They also seem to have concluded that policing is a very different activity from intelligence work, which is not necessarily concerned with either arrests or law enforcement. Intelligence-gathering involves acquiring information in an anticipatory, prophylactic manner – fragments of information from here or there which may or may not become important one day. This distinction between intelligence and policing continues to the present day, and in fact is one of the reasons why the FBI at the start of the twenty-first century is considered to be ill-equipped to deal with the threat of terrorism, which requires anticipatory intelligence, not policing.¹⁰

Despite the notable continuities in MI5 and SIS’s history from 1909 onwards, it would be erroneous to suppose that in the years immediately after their foundation they were anything like the services they would later become. It is a myth that ever since the sixteenth century ‘British intelligence’, like the British empire itself, grew steadily in size and influence, spreading its tentacles across the world. This myth was mostly derived from Edwardian spy novelists like Erskine Childers, whose writings, such as Riddle of the Sands, depicted a powerful British intelligence service actively thwarting its enemies both at home and abroad. The reality could not have been further from such fictions. For years after their establishment, both MI5 and SIS remained desperately short of resources. The diary of MI5’s first and longest-serving head, Sir Vernon Kell, reveals how perilous the organisation’s existence was in its early days. In the years before the First World War, MI5 and SIS were both run on shoestring budgets. At the outbreak of war in 1914, MI5 had an entire staff of just fifteen – including the office’s caretaker. Staff numbers in SIS were similarly meagre. During his first week on the job in 1909, the first Chief of SIS, Sir Mansfield Cumming, noted rather miserably in his diary that he sat alone in his new office, without the telephone ringing, and with no one visiting him because the Bureau was too secret to be listed in a Whitehall telephone directory. This was very much a one-man-and-a-dog operation.¹¹

The First World War led to the massive expansion of the machinery of Britain’s secret state – just as it did in all other major belligerent European powers. In fact, it is fair to say that the war was the event that created the modern natio...

Table of contents

• Title Page
• Praise for Empire of Secrets
• Dedication
• Epigraph
• Contents
• List of Illustrations
• Abbreviations and Glossary
• Map: Principal MI5 posts in the empire and Commonwealth in the early Cold War
• Introduction
• 1. Victoria’s Secrets: British Intelligence and Empire Before the Second World War
• 2. Strategic Deception: British Intelligence, Special Operations and Empire in the Second World War
• 3. ‘The Red Light is Definitely Showing’: MI5, the British Mandate of Palestine and Zionist Terrorism
• 4. The Empire Strikes Back: The British Secret State and Imperial Security in the Early Cold War
• 5. Jungle Warfare: British Intelligence and the Malayan Emergency
• 6. British Intelligence and the Setting Sun on Britain’s African Empire
• 7. British Intelligence, Covert Action and Counter-Insurgency in the Middle East
• Conclusion – British Intelligence: The Last Penumbra of Empire
• Picture Section
• Note on Sources and Methodology
• Notes
• Bibliography
• Index
• Acknowledgements
• About the Author
• Copyright
• About the Publisher