Snowball in a Blizzard

The Tricky Problem of Uncertainty in Medicine

Steven Hatch

• Publisher: Atlantic Books
• Year: 2016
• ISBN:
  9781782399889
• Topic: Medizin
• Subtopic: Medizinische Theorie, Praxis & Referenz

There's a running joke among radiologists: finding a possible tumour in a mammogram, they say, is akin to finding a snowball in a blizzard. The result? Up to 30% of breast cancer surgeries are done on those who have no cancer at all.

In this landmark book, medical professor Steven Hatch reveals that although modern medicine has reached new levels of scientific prowess, we know far less than we think we do. Indeed, CAT scans, MRIs, Mammograms, and blood tests provide a wall of data where false positives are rife. Thus, to be a good doctor, surgeon, or psychiatrist, it is just as important to know what one doesn't know, as what one does.

Covering everything from the efficacy of Prozac to the regular barrage of health advice by the media (e.g. bacon causing cancer), Hatch shows why it's essential that doctors and their patients know how to interpret data. A drug that might be very effective to a certain cohort of patients suddenly becomes little more than a placebo when given to large bodies of the population (think Statins). A prognosis of 10 years to live might be accurate on average but mean nothing for the individual cancer patient. Filled with the kind of revelations about flawed human reasoning that made Daniel Kahneman's Thinking Fast and Slow a bestseller, this is the must-read medical book for 2016.

1

PRIMUM NON NOCERE: THE MOTIVATIONS AND HAZARDS OF OVERDIAGNOSIS

Many people are overconfident, prone to place too much faith in their intuitions . . . when people believe a conclusion is true, they are also very likely to believe arguments that appear to support it, even when those arguments are unsound.

—DANIEL KAHNEMAN

Stripped of all its complexities, the process of medicine consists of two basic activities. The first activity concerns diagnosis: the identification of some condition, some malady that afflicts the body. The second activity—done with some hope and luck in addition to some science—is treatment: the attempted eradication of disease, or at least the relief of symptoms, through the prescription of medicines or the correction of anatomy via surgery. Doctors, and the folk healers who preceded them, have been engaged in this two-pronged approach for millennia.

The emphasis on careful diagnosis characterizes the vocation of medicine even from antiquity: it is why we still regard Hippocrates, who lived more than 2,000 years ago, as our profession’s paterfamilias. Although our medicine has changed radically since the time of the ancient Greeks, we can still see the faint outlines of their diagnostic approach to this day. For instance, medical students routinely learn to observe a characteristically odd shape to fingernails known as “clubbing”; its identification suggests a variety of chronic diseases ranging from emphysema to cancer, and Hippocrates himself is credited with the earliest known clinical descriptions of the phenomenon. To this day, some of the old-timers will still use the term “Hippocratic nails” to describe clubbing.

The process of diagnosis until very recently consisted mainly of talking to patients about their symptoms and examining them—that is, investigating their bodies for the physical signs associated with disease. But we have witnessed a quantum leap over the past generation or so, and the act of diagnosis today would have blown the mind of a doctor from one hundred years ago. What doctors possess now to not only identify but precisely locate a disease is astonishing. MRIs and CT scans allow us to peek under the hood, so to speak, and look all the way through the body as if we had sliced it up, sometimes finding small pockets of infection or cancer that would have been difficult bordering on impossible to find before their advent. Blood tests can distinguish among dozens of viruses and bacteria, identify whether some infections occurred in the past or are happening in the present, report our electrolyte levels, locate the exact spot of a genetic mutation out of 3 billion “letters” of DNA, and do all of this within days, if not hours. Cameras can now be used to visualize seemingly any part of the body on a television screen, even to the point where we can swallow a pill with a tiny camera and in less than a day drop off a cartridge with an image file of a colonoscopy.

As we launch into this first stop on our tour of uncertainty, one point requires some emphasis at the outset: this is amazing. We are living in a golden age of diagnosis. There are entire categories of disease of which we have only recently become aware—conditions that we had mistakenly lumped into others, lacking the technology to distinguish them. As we improve in our ability to discover diseases and find ever-better tests by which we can diagnose such diseases, the likelihood that doctors will make life better for their patients increases exponentially. These new technologies are absolutely indispensable to that process.

That said, the changes wrought by what we might call the great diagnostic shift is only beginning to be understood. There has been a slow recognition that not all of these changes have been beneficial. The precision with which we can make diagnoses is profound, but precision is not the same thing as certainty. One of the main problems with diagnosis today is that these new technologies are so sensitive, so able to find diseases, and in such early stages of development, that they leave us too confident that we’ve identified disease when we probably haven’t. This process of finding-disease-that-isn’t-disease is called overdiagnosis, and it is possibly the most important real-world consequence of our misguided faith in certainty.

The best place to start to illustrate this, ironically, is by investigating a profession that still barely uses all of this new diagnostic technology: psychiatry. I’ll begin by looking at one of the more infamous chapters in the history of that profession and considering how that might be related to CTs and MRIs and blood tests and all the rest.

Error Machines

Between 1969 and 1972, a group of eight patients had presented to a variety of psychiatric hospitals across the United States with a remarkably similar complaint. They all said they were hearing voices. This is hardly uncommon in psychiatric institutions, but in each case the voices said the same three words: “empty,” “hollow,” and “thud.”

Many of these eight patients worked in the field of mental health: four were either psychologists or psychiatrists, and a fifth was a graduate student in psychology. The remaining three included another doctor (a pediatrician), a painter, and a housewife. None of them had any prior history of psychiatric problems, and none of them behaved strangely during their initial evaluations. Following the initial assessments, seven of these eight patients were admitted with the diagnosis of paranoid schizophrenia, while the eighth was diagnosed with manic-depressive psychosis.

What made this cohort of patients singularly interesting to the profession of psychiatry, and why they are known to posterity, is that in addition to having precisely the same auditory hallucination at different times and in different places, they shared one other feature: they were all completely sane. These eight “patients” were, in fact, volunteers for a brilliant and devious study conducted on the psychiatrists, and more broadly on the staff at the psychiatric facilities where the evaluations took place. The results were published in 1973 in one of the most unusual papers ever to grace the pages of Science, then as now one of the greatest scientific journals in the world.

The research question of the author, Dr. David Rosenhan, was elegant in its simplicity: Could highly trained mental health professionals detect a sane patient in an insane place? The answer was a resounding no. The details of what has since been come to be called the Rosenhan experiment make for somewhat uncomfortable reading and can be found without difficulty on the Internet. The results, written in a fluid prose style easily comprehensible to nonscientists, should be humbling to the profession of psychiatry. More than forty years after the publication of this seminal experiment, I am still not sure if that is the case.

The key to the experiment relied on the distinction between a psychiatric symptom in a sane person and the clinical definition of insanity. For the experiment, the participants were otherwise normal people presenting with one single, transient, unusual symptom. They all reported experiencing an isolated auditory hallucination, and although such a symptom is associated with mental illness, a lone hallucination is not equivalent to schizophrenia. Many people experience transitory hallucinations without having a total psychiatric breakdown. What is required for a diagnosis like schizophrenia is evidence of a person’s complete inability to function in the world as a consequence of the hallucinations and the inability to stop or control such hallucinations to the point where that person is overwhelmed. The pseudopatients—which included Rosenhan himself—were all instructed to behave as they normally do in life, which prior to their admission had raised no alarm to anyone. The only psychiatric diagnosis that could reasonably be made fell far short of schizophrenia or manic depression.

Their sanity made no difference. Despite the paucity of evidence that they suffered from deeper pathology, not only did all of the participants end up with the most global and profound labels of mental illness, they were held for astonishingly long times to manage these conditions. The average length of hospitalization was nineteen days, with the shortest lasting seven days and the longest a dumbfounding fifty-two. Moreover, it was hardly the psychiatrists alone who failed to notice the sanity of the participants, as the nurses and attendants treated the patients in the same manner. Indeed, a review of the nursing records showed that they interpreted specific behaviors as evidence of pathology, even though such behavior would raise no suspicions outside a mental hospital. “Patient engaged in writing behavior” was one of the observations made when one pseudopatient was simply taking notes on his surroundings for later review, a line that has since been quoted hundreds of times in the medical literature and by patient advocates skeptical of modern psychiatry’s benefits.

By contrast, the insane—that is, the true psychiatric patients—were surprisingly good at spotting sanity. During the first three hospitalizations, 35 of 118 patients voiced varying degrees of suspicion that the participants were either journalists, professors, or some other kind of sane person “checking up on the hospital.” Despite reassurances that they “had been sick before but were fine now,” some of the actual patients persisted in their belief that the pseudopatients had no history of psychiatric problems for the entire length of the hospitalizations in question. In some ways, the number is even more remarkable, because the 35 “diagnoses” offered by the psychiatric patients came unbidden, in contrast to those of the psychiatry staff who were charged with the responsibility of making a diagnosis. “The fact that the patients often recognized normality when staff did not raises important questions,” Rosenhan commented in what can only be regarded as a major understatement.

Yet the most fiendish portion of the Rosenhan experiment came after these pseudopatients were discharged. Rosenhan presented the results to the staff at a prestigious psychiatric research and teaching hospital and was greeted with widespread disbelief. Surely, the staff argued, such a miscarriage of medicine would not occur at our institution. So Rosenhan tipped his hand and informed them that over the course of the next three months, one or more pseudopatients would be admitted to their hospital. He provided the staff with questionnaires to assess their assessments, so to speak, of the level of sanity or insanity of the admitted patients. During the experimental period, 193 patients were admitted. Of these, 41 were considered to be pseudopatients by at least one staff member, and 23 were labeled pseudopatients by the psychiatrists. Nineteen patients in all were identified as pseudopatients by at least two mental health professionals. In reality, none of the 193 admissions were pseudopatients—or as Rosenhan sardonically noted, if they were, they weren’t involved in his research.

Viewed in sum, the Rosenhan experiment appears to be a case study in the limitations in psychiatric diagnosis and the effects of institutionalization on the mentally ill. “It could be a mistake, and a very unfortunate one, to consider that what happened to us derived from malice or stupidity on the part of the staff,” he wrote in conclusion. “Where they failed, as they sometimes did painfully, it would be more accurate to attribute those failures to the environment in which they, too, found themselves than to personal callousness.”

The meaning of the Rosenhan experiment is still hotly debated to the present day. Does it demonstrate that psychiatry is hopelessly mired in subjective impressions, where once a label is applied, it sticks beyond all reason, to the point that prolonged sane behavior cannot even be recognized for what it is? Or was the experiment jerry-rigged to arrive at this conclusion in the first place? Among the criticisms of the research was that, instead of testing the ability of the psychiatrists to diagnose insanity, it was actually just testing the ability of the patients to lie. One analysis offered the following counterfactual: if a person could steal a few pints of blood, swallow it, and later present to an emergency room vomiting up the blood without explaining what happened, the subsequent diagnosis of a stomach ulcer wouldn’t mean that the staff didn’t know how to diagnose that condition.

My goal here is not to take a strong side in this dispute, although on the whole I do find Dr. Rosenhan’s critique compelling and think the data does indeed support his thesis. Regardless, I wish to point out that the legacy of the Rosenhan experiment is almost universally understood through the lens of psychiatry and what it means for that profession, and nobody ever considers the wider implications it may have for the rest of medicine. Psychiatrists are often, if only politely and subtly, dismissed by their colleagues for engaging in what is seen as a quasimedieval enterprise. Internists, surgeons, and obstetricians can all employ tests that provide binary or quantifiable results. Their ability to diagnose conditions is vastly superior to the crude tools of psychiatry, the reasoning goes. Why should they be surprised that someone showed psychiatrists, however bright they are, don’t know what they’re doing?

But I would argue the experiment’s results can in fact be applied well beyond the realm of psychiatry—they can be generalized to all doctors when they consider all diagnoses. And there is a growing body of research showing that nonpsychiatric doctors can be just as flawed in their judgments as insanity doctors who are unable to see sane people standing right in front of them. Rosenhan thought he was doing an experiment on diagnosis in psychiatry, but at a deeper level he was performing an experiment on diagnosis itself, and it wasn’t psychiatrists per se who failed in this regard, it was doctors. The fact that very few physicians outside of psychiatry who have read this research think it applies to them speaks more to the provincialism of these specialties than to anything else. As we shall see, there is increasing evidence that not only are all physicians prone to the same kinds of cognitive errors witnessed in the Rosenhan experiment, but there is a good chance physicians are doing harm as a result.

This book is primarily concerned with data. As I will try to show in the chapters to come, most data in the realm of medicine is inherently fuzzy—tests are almost never 100 percent positive or 100 percent negative, and there are real-world consequences related to just how certain one can be that a test’s result matches the underlying reality. We are only very rarely at the far end of the spectrum of certainty, where we can be extremely confident of our diagnoses and the benefits of our treatments. Some diseases require only a single test whose results can be interpreted with high degrees of certainty, but many others require a careful consideration of several pieces of information, at least some of which can be contradictory.

The human brain making a diagnosis can be thought of as another illustration of this principle. The psychiatrists in the Rosenhan experiment made two kinds of diagnostic errors. First, they diagnosed insanity in patients who were not insane: we call that kind of diagnosis a “false positive.” Likewise, when tipped off that pseudopatients would be wandering the halls of their institutions...