Research in Crisis

Blueprint to Overhaul the Broken Knowledge Factory

Les Coleman

• Publisher: Routledge
• Year: 2020
• ISBN:
  9781000172164
• Edition: 1
• Topic: Business
• Subtopic: Ricerca e sviluppo
• Index: Business

This book explores the weak explanatory and predictive power of theories across disciplines, explains reasons for limited expertise after centuries of scientific effort, and sets forth strategies to accelerate knowledge and manage a future we can only dimly comprehend.

Gaps in knowledge arose because common, natural and artificial phenomena are fundamentally hard to understand, and in expertise persists because research is unproductive. This book argues that weak research comes with huge opportunity cost because it stymies optimum decision making by government, corporations and individuals. Research needs restructuring which must come from governments' top down requirement that funding bodies foster applied research with real-world impact, and that universities influence scientific publishers to improve their publications' integrity.

This book seeks to catalyse extinction events for theories in most disciplines, which would clear a path for solving multiple crises in research. The author cautions that this process would be disruptive, unpopular and painful.

1 What this book is about

When the legend becomes fact, print the legend.

Publisher Maxwell Scott in Ford (1962) The Man Who Shot Liberty Valance

The wicked flee when none pursueth.

Proverbs 28:1

Researchers across the natural and social sciences are too confident about their knowledge. Theories can barely explain the recent past, and are all but blind to the future which is why its risks are so high. This book seeks to help make scientific research more productive. It critically evaluates the causes of poor research productivity and proposes an integrated solution that relies on targeted incentives and rolls out better techniques that are already proving their merit. The objective is to boost productivity of the research factory so its output has real-world applicability (Figure 1.1).

Figure 1.1 Essentials of Research in Crisis.

About $2 trillion is spent globally on research each year, and its weak outcomes matter because good research is the only source of our scarcest commodity which is knowledge about the future. Standing beside a supercharged volcano, investing in an overly exuberant market or not recognising a growing cancer is risky because we do not know their futures. Also, the gold standard of evidence-based decisions requires knowledge about the future, and its absence explains every failed policy or strategy. Finally, complex problems such as poor quality of life for indigenous people, climate change and modern epidemics of obesity and mental health remain unsolved because knowledge is not strong enough to secure consensus on appropriate remedial actions.

Knowledge about the future of any variable or system is only possible if its underlying mechanism is understood, so good theory is essential to reliable predictions. Unfortunately predictability is lacking except for mechanical systems such as planetary orbits or simple cause and effect such as aspirin-cures-headache. According to one wag, we can predict the moment of an eclipse, but not the decade of a revolution.

There is no doubt that we have powerful technical skills, and today’s technologies are marvellous. But we still lack the ability to explain most of the world around us and make meaningful predictions. The reason why is that explaining and predicting a dynamic system require theory that is robust to changes in time and setting, whereas theories of even the most common phenomena merely model a narrow record of the past. Most theories are no more than tautologies that quantify hindsight or state the obvious.

Researchers cover up this situation by ensuring theory never contacts the real world. When avoiding reality becomes impractical, expediency sees discrepancies between theory and observations bridged by inventing something that is unknown and unseeable to plug the gap. Thus dark matter dominates not just cosmology but also earthly affairs as diverse as markets, decisions and medicine. These faith-based descriptions rarely meet the requirements of government and industry for accurate predictions of phenomena in different environments and times. Thus the tangible contribution of theoretical research is limited despite its impressive imagery.

It matters that research is weak. First, because risk is high when we lack knowledge, so poor understanding of the natural and man-made world contributes to its dangers. Second is that – because reliable forecasts are impractical – decision makers must make do with only a dim idea of what the future holds: policy based on sound theory and empirical evidence is a chimera. Finally, when knowledge and theories cannot add value, they require their own extinction event to clear the way for a new approach.

I have always despised people who criticise but cannot offer a realistic and acceptable solution, and so have provided a set of recommendations to repurpose research. These are based on two planks. One is to ensure research institutions and incentives are unconditionally committed to constructing workable theories. The other is to eschew thought bubbles and only advocate solutions that are already in place and working (even if only on a small scale). This points to proven ways to find scientific truth that can solve problems and reduce risks. The aim is to redirect research strategy at both the institutional and individual levels to accelerate and manage knowledge accumulation and serve as a facilitator of change and risk reduction.

This book is based on three contentions. One is that lack of expertise makes it hard to unravel the world around us, so we have trivial forecasting capability and face chronic uncertainty and puzzles about common events. In short, we live in an age of inexpertise. The second is that weak knowledge arises from inadequate theory, where theory is defined as a system of ideas that explains something. A third contention is that solutions to weak research lie in the hands of those who administer its finances. Research councils or science foundations provide most research funding and resources, which gives them the clout to force strategies on researchers that will improve their conduct and productivity. Universities provide most funds to scientific journals which are the main conduit of research, and can force better business models and publication of rigorous applied research.

Although concerns such as ignorance, uncertainty and weak theory have been a marginal, neglected topic in scientific research (Smithson, 1989), a growing number of scientists now ponder the real-world relevance of research. With remarkable consistency, knowledge-based disciplines are facing scepticism through analyses with titles such as ‘Why most published research findings are false’ by medical researcher John Ioannidis (2005). Contributions have become mainstream, including Ian Cheney’s 2018 documentary entitled ‘The Most Unknown’ which interviews nine scientists about the limits of knowledge.

Research in Crisis builds on existing work and is intended for innovators in government, universities, business and the community who seek a reimagined research paradigm that will slingshot its productivity. This book explores linkages between research, theorising and forecasting. More particularly, it unravels the puzzle of how we can have invested so much treasure and labour in research over centuries and have so little to show for it in terms of practical understanding. Modern skills and technologies give researchers the ability to observe and analyse our world in unprecedented detail. Their successful application to research should predict future conditions and – by avoiding risks and maximising opportunities – improve social and economic decisions. But, no matter what question is involved, humans lack expertise in the sense that they cannot explain what they see or meaningfully describe future conditions.

This book innovates by envisaging important phenomena as complex systems whose multiple, time-variant determinants are embedded within an intricate framework, and thus pose a formidable challenge to understand. Research, though, socialises knowledge by taking a snapshot that is examined out of context and ignores phenomena’s setting and real-world behaviour. It assumes systems are in equilibrium despite their obvious variation over time and interprets data through normative paradigms, which are idealised conceptions of how phenomena should behave. Now, though, no discipline can explain more than about five to ten percent of what we see: dark matter does not just fill the cosmos, but – in the guise of behavioural finance, the sub-conscious and similar – is the most common explanation of puzzles in our world.

To understand why important phenomena are so hard to decipher, consider a scientific apparatus in the style of a pitching machine that propels balls of known mass at known speed and trajectory across a standard playing surface. Start with a billiard ball. The scientist will know the decelerating effects of the table’s surface and cushions and be able to fairly accurately predict the resting place of any ball. This is an open-loop system where all the required data are available in advance; and it is intensive in that it has a single, readily observable component. Now move to a football pitch (soccer field for North American and Australian readers) where 22 players watch the apparatus being readied and position themselves to move when the ball lets fly. This is a closed-loop system whose parts (that is, the players) respond to future data. As the ball moves, players favour their team by intercepting it or their opponents and thus modulate passage of the ball. Players and referee also react to players’ actions and to the ball’s movements, which induces feedback. Thus the ball’s path is extensive as the aggregate of a variety of complex sub-systems (decisions and actions of the players and referee), and its final resting place in one of the nets or outside the field is unknowable. After the event, it is reasonably easy to explain the physics of the ball’s path, although not the players’ underlying motives or intent. But the path could not be meaningfully predicted because player decisions are only dimly understood, and feedbacks – such as an interception or goal – have not yet occurred. The best a researcher can do is to run multiple simulations and identify the distribution of outcomes. Subsequent forecasts are better than a guess, but individual results still look random from human perspectives and time scales.

Reality is that our world is dominated by dynamic, closed-loop systems that behave in the style of minimally predictable football games. They come in many forms. Some systems can be codified and/or measured such as contests of skill; others may be contained and identifiable such as a human heart, or be a broad, opaque aggregate such as an economy. Systems can be natural such as weather and earthquakes, or involve sentient participants such as human-designed companies, professions and political processes. These systems unfold in line with feedbacks that respond to outcomes, especially other participants’ decisions. Causal effects are often obscure or hidden, and can be so extensive and varied as to defy compilation; and systems’ progress depends on future developments for which data are not yet available. None of these everyday systems reaches more than brief equilibrium, and they are profoundly hard to come to grips with.

This chronic uncertainty sits uneasily with mankind’s thirst for convincing explanations of the world, with the expectation that – unlike beasts – humans do not have to suffer nature and inexplicable events, but can describe and shape them. Victor Frankl (1962) in Man’s Search for Meaning argued that meaning “is the primary motivation of his life … It is unique and specific in that it must and can be filled by him alone”. For Frankl, frustration of the natural will-to-meaning of life and the resulting absence of a personal, specific concept of it leads to an existential vacuum and neurosis. This complements Nietzsche’s concept that we seek to know the why of life so we can make sense of it.

The search for meaning explains centuries-old notions of logical discourse and rationality. It makes truth a social construct and leads researchers to emphasise their hosepipe of knowledge. When researchers encounter difficulty in understanding systems or make puzzling observations that challenge existing paradigms, they commonly paper over cracks to preserve the facade of meaning. One solution invents something that cannot be seen, but must exist to make sense of puzzling evidence. Cosmology offers the most obvious response to a knowledge deficit. Many tens of billions of dollars have been expended and at least 20 astronauts have died in mankind’s quest to understand the universe as described by Einstein’s theory of general relativity and Newton’s laws of gravity. There is, though, only enough visible matter to explain five percent of the universe’s behaviour, and its missing mass problem is solved by hypothesising the existence of dark matter and dark energy which are quite unknown except for their gravitational effects. This follows a well-worn – and generally unsuccessful – path for physicists who invent mythical objects to resolve puzzling observations. Many examples involve invisible substances such as a resisting medium in space that slowed the period of Encke’s comet during the 18th and 19th centuries, and aether that was proposed in the 19th century as the medium through which light is transmitted. Other solutions involve various undetected cosmic bodies that explain perturbations in planetary orbits.

Cosmology, though, is not alone because dark matter serves as an analogy for the response by puzzled researchers in many fields. Chemistry has a similar track record, which – in the 18th century – saw identification of phlogiston (a substance released in combustion) and caloric (a substance which flows from hotter bodies to colder bodies). Economists of that time postulated an invisible hand that guides markets, and more recently invented behavioural finance to explain decision biases and price anomalies that stymie modern portfolio theory. Dark matter makes up much of our bodies, too: 90 percent of decisions are made subconsciously; idiopathic disease makes sense of difficult-to-explain appearance of symptoms, while the placebo effect rationalises their difficult-to-explain disappearance.

It is important to recognise that many now rejected theories once provided satisfactory explanations after discovery by leading scientists of their time, using techniques that are not much different to those of today. Centuries of evidence show that the scientific method produces rubbish theories and is not associated with good science.

Although dark matter solutions do not prove robust, they are prevalent in cultures with an imperative to identify order and meaning in the world. These societies resolve their existence by inventing an unseen cosmic-scale gardener,¹ which encourages popular wisdom and science to swallow dark matter, aether and other undetectable explanations for hard to comprehend phenomena. It is no coincidence that theory and theism have a common Greek root, respectively, being sourced from theoria meaning speculation and theos meaning god.

Dark matter solutions prove irresistible because they cannot be disproven. For instance, a recent article in the prestigious Physical Review Letters opened with the observation that “an abundance of astrophysical observations suggests the existence of a non-luminous, massive component of the universe called dark matter”; but then reported that a year of searching for dark matter using a huge particle detector proved to be another of many failures (Aprile, Aalbers et al., 2018). For me, Occam had the right idea, which is that what...