The Limitations of Social Research

M. D. Shipman

• Publisher: Routledge
• Year: 2014
• Print ISBN: 9781138469785
• eBook ISBN: 9781317883722
• Edition: 4
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

'Does the evidence reflect the reality under investigation?' This is just one of the important questions Marten Shipman asks in the fourth edition of his highly successful book, The Limitations of Social Research. Substantially revised and up-dated it probes not only the technical stages of research, but also its assumptions, procedures and dissemination.

PART ONE

Approaches to Social Research

CHAPTER 1

Social research

Our ancestors, faced with a problem, were likely to consult a priest, an oracle, some hopefully wise old neighbour, custom, holy writ. That reference to authority persists. But we are as likely to engage in or look to research. Here the concern is with research that is published. That still leaves an overlap with filling in a tax form and peeking over the garden fence. Further, that overlap is a clue to the limitations that are the subject of this book. Social research is about humans investigating humans. That raises philosophical as well as technical problems. It also means that the motivations to research are often mixed up, particularly the urge to discover new knowledge, to benefit humanity and to raise awareness of injustice.

There are then many routes to understanding the human condition. The mystic and the priest have their own maps. These are often private, based on some external source and jealously guarded. Authority is not negotiable. The authority of research as a claim to knowledge is distinctive because it rests on publicity not privacy. It is disclosure that enables others to examine and assess the work and hence establish its authority. This is a social activity dependent on scientific communities that are organized to assess, approve or reject without deference (Popper 1945). But this cannot be exclusive. Social research is reviewed by government, professions and the lay public who bring their own informed if distinctive criteria to bear to assess the evidence for use. They need sufficient information to be able to do this. But researchers can leave out essential details from published accounts. Peer review can be inadequate. Claims for research as a source of knowledge can be as private as those from any cabala.

The claims for research as a path to knowledge

Research is a public exercise

Secrecy belongs to the era of arcana, when knowledge was produced in communities of priests or scholars jealously guarding their sources from the public. The alchemist, the quack, the mystic, the priest, claimed privileged access to the truth but denied it to outsiders. The distinguishing feature of research is that it is published so that it can be reviewed by peers. Only then is a claim to truth accepted or rejected. The claim to knowledge is based on surviving criticism, not by reference to mysteries beyond human reason.

This is the most important claim. Publication should enable readers to assess credibility from initial conjecture to final conclusion. Only when sufficient detail is made available can strengths and weaknesses be assessed. That is not only essential for the peer review that decides reliability and validity, but for the professional and lay-users of the evidence. It is the major focus of this book because publication can often conceal rather than reveal. Conventions of reporting today can be as secretive as the mumbo jumbo of alchemists. The researcher is often concealed behind third-person language. The published reasons for doing the work this way, selecting that data, analysing the results like this, drawing those conclusions are rarely sufficient for those outside the often small research community involved to make an informed judgement. Those inside often have to use knowledge that is not in the public domain.

Research makes explicit claims to knowledge

The second characteristic of research is that from start to finish its procedures are claimed to be explicit. To count as research, enquiry has to be more than the collection of prominent features of an interesting topic as they strike the inquisitive. It has to have defined and restricted aims if it is to be a claim to knowledge. That is often achieved by using models that simplify the confusion of the world by concentrating on a few features related by a theory. The research design and its rationale will be spelled out and justified. The analysis of evidence and the conclusions drawn will be referenced to the aims and methods.

This does not rule out changes in direction as research progresses. Indeed, in fieldwork described in Chapter 3, this is common as new leads look promising. But it does mean that this should be planned for and made clear in reporting. It is legitimate to raise awareness or promote change, but not published tangled up with a search for knowledge under the common title of research. The implicit cannot be assessed. There should not be any need for the reader to puzzle over what was really being investigated, how this was done and why.

The explicit nature of research is most marked in its systematic procedures. There is a design and progress recorded in fieldnotes. It is thought through, carried out, recorded and reported in ways that can be justified and are made public. The design will have been chosen because it is suited to the issue being researched. But it will also be chosen on the basis of assumptions in the discipline of the researcher. It is a discipline because its members share views on the nature of the world, how we can get to know about these, what limits there are to the knowledge produced. There is both method and methodology because the various approaches are rooted in different views of the world that is being investigated.

The responsibility for ensuring that research is public, explicit and systematic lies with the researcher. This may be difficult. Time and resources may be limited. There are constraints on space in publication. But there are too many examples in this book of influential but flawed evidence for this responsibility to be ignored. There is also a responsibility on the readers of research. If evidence is to be used it has to be assessed. There is never complete agreement among peers. Further, if reality is indeed what is defined as real, each reader is likely to review research differently. The claims are based on exposure to review. The more perspectives that are brought to bear, the less likely it will be that faulty theories will be developed and misleading practices promoted. Further, the greater the number engaging in review, the more likely is the flawed or fraudulent to be exposed.

Thus there is particular emphasis in this book on public disclosure as central to all enquiry titled as research. There is also the assumption that the reader will adopt the sceptical stance that ensures that any published work, including this book, will be critically assessed as it is read. The guiding principle comes again from Popper. There must be no deference in the scrutiny. That is the condition of the open society and the accumulation of knowledge (Popper 1945). Unfortunately the conventions of publishing tend to exclude the crucial first stages of research that are in the biographies of the researchers and the models used in their specialism. This often missing information accounts for the style of research. It can be scientific, trying to reduce the effects of extraneous factors. It can be freeranging, designed to follow leads that look promising. The evidence collected can also be used to try to remedy injustice. But the bottom line is that if it is under the title research it should be public not private, open to review not a mystery known only to a closed group.

Behind these two basic features of research lies the communal organization of science. The lone alchemist jealously protecting results from spies was doomed once scientists organized for communication and review. The Royal Society and its French and Italian counterparts, established in the seventeenth century, gave a seal of approval to science as a profession and provided a forum for public debate and review. Today academic libraries are bankrupted by the proliferation of journals to facilitate the publication that informs colleagues, attracts money, advances careers, brings prestige to institutions and adds new knowledge. It is now so specialized that there are barriers to the necessary communication. But there is a vast array of users of social research capable of reviewing critically. Too often they are denied the chance.

There is one more necessary comment before looking at research within social science. Scientists are a varied lot, not just in their range of subject specialisms, but in the way they go about their work. Medawar (1986) distinguishes two conceptions of science. The romantic conception is of a speculative excursion into the unknown. It is poetic, imaginative and should not be constrained. In social science this is often found in the interpretive research described in Chapter 3. The alternative conception is that truth lies in nature and must be extracted, examined and judged on the evidence. That scientistic social research is discussed in Chapter 2. But it remains a creative, human, often funny and hence fallible activity. Here then is the unhappy case of the discovery of N-rays.

The discovery of X-rays by Röntgen in 1895 led to a search for other rays with other wonderful properties. In 1903, Blondlot announced the discovery of N-rays. By 1904, 77 scientific publications described their detection, properties and applications. But there was a mystery. Outside France nobody could find them. One explanation from inside France was that Anglo-Saxon senses had been dulled by exposure to fog and beer. Then a sneaky American visiting Blondlot’s laboratory removed a crucial lens from the apparatus during a demonstration, yet the N-rays continued to be observed. The N-ray disappeared from science and poor Mr or Professor or Prosper, or René Blondlot, Blondot or Blandot, recipient of either the Lalande or Leconte Prize, went mad. Even in death he has not been respected for all these versions of his name and his prize appeared in the references used here (Good, 1962; Eysenck, 1965; Firth, 1969; Broad and Wade, 1982).

The human subject in social research

The step from natural to social or human sciences such as economics, psychology and sociology, means that the subject is now thinking humans and their transactions. They will be learning from and acting on the research situation. Social researchers are also part of the world they research. Their activities provide clues for those they study. Across the years humans in the social sciences have been modelled as driven by instincts, drives, the id, needs. They have been modelled as actors playing roles written for them, or as making roles, as manifesting rational expectations, as information-processors, through to enquiring scientists. Human behaviour is still modelled by experimenting with apes, pigeons and rats at one extreme and by comparisons with gods at the other. Each metaphor or model leads to different ways of defining the human condition and hence of designing and reporting research. The questions asked will be varied and many answers will not even be comparable.

In this book it is assumed that humans are active, enquiring, interpreting, trying to make sense of their world. Thus researched and researchers are both modelled as trying to find out and understand what is happening. Both bring to bear their personal, political as well as professional beliefs to give meaning to events and take action. All are members of groups that share beliefs, ways of finding out and criteria for judging validity. Through those groups they gain prestige and influence others. Beware, for questions from such a popular model lead to optimistic answers. Those who assume that life is nasty, brutish and short reach different conclusions.

The distinguishing marks of professional researchers are that their work is focused and made public for review. But they too belong to groups, disciplines and specialisms where they learn to use, develop and validate models. They ‘see’ the world through these models. They depend on colleagues to judge their work and hence to determine their careers. These research communities can be tight-knit, with membership, reputation, qualification and promotion, as well as the assessment of research, controlled by a few authoritative figures. They can hold in common views that are openly political. They can also be loose associations of buccaneers. But the individual professional is still influenced by the group through which their work is assessed and their career determined.

The importance of assumptions about the human condition lies in determining the questions asked and in conjectures about answers. Experimental methods, or approximations to them, are appropriate if humans are modelled as the product of stimulus-response bonds. Other ‘scientistic’ methods may be legitimate if humans are modelled on the computer or as information-processors. But if the model is of humans who are always working out and sharing new meanings of the world around them, a social science has to be ‘interpretive’, in order to study those shared meanings. That subjective world seems self-evident to those involved. That means getting into the field without disturbing it and looking and listening, until the lay meanings are understood in lay terms. There can be no imposition of meanings through hypotheses formulated in advance.

It has not been easy for social scientists to give up the idea that there were laws of human behaviour to be found equivalent to those in the natural world. It must have been gratifying to claim you knew why and hence could predict future developments. But humans are slippery subject matter. It is reasonable to assume that beyond simple knee-jerk reactions, little is predictable in human behaviour, especially over time or across cultures. Humans think, particularly about what the researcher is doing. There have been two major responses to the arguments against a science of human behaviour. Some social scientists, particularly in sociology, rejected science as an appropriate method. Others redefined it to accommodate the study of the human condition.

Once the change of model of the human switches attention from the experiment, the test and tick-the-box questionnaires, to observation in the field, classroom or bedroom, the opportunity for fun increases. The models now overlap with those in the arts. The divine comedy, the naked ape, the noble savage, the concrete jungle, all the world’s a stage, paradise lost, all produced new insights. The social sciences have as much variety. From here on this will be reduced to the conventional distinction between the scientistic and the interpretive. Both of these approaches to research can be used to go beyond informing to reforming, emancipating. But this is a gross simplification. Each of the many metaphors leads to specialization within social sciences and hence to different research questions, designs and results. That is realistic. Human life is varied and unpredictable. The grand theories, the metanarratives and the systems of much early social science, telling how human society developed, are now recognized as fallible. Contemporary social research tends to aim at revealing the variety and the diversity of human life. But that revelation, if it is through research, still means explicit definition, systematic enquiry and full disclosure of procedures and results for peer review.

Social researchers at work

There is a continuing demand for research. The Economic and Social Research Council spent more than £40 million of research and nearly £20 million on research training in 1995–96. The explanation lies in the human concern to give meaning to events. We need to know. We also need to believe that we can know. That questing marks out researchers. They are imaginative and adventurous. It is tempting to describe social research just as techniques for collecting information. Even accounts that show how methods are related to their underlying assumptions tend to concentrate on two models: one, scientistic, sharing the assumptions of natural science, the other interpretive, aimed at understanding and building on the way humans make sense of their world. Yet that is an impoverished version of a varied and exciting occupation. From fact-gatherers to consciousness-rousers researchers exercise a variety of skills in pursuit of a variety of objectives.

Much of this book is about the contrasts between these two research traditions. Yet both can be used not just to study but to expose injustice and hence suggest how to improve the human lot. There is a long and honourable tradition of such critical research. The limitations of social research can only be appreciated if the intentions as well as the procedures of the researcher can be assessed. The justification for a claim to research lies in it being restricted in its ambitions and a public exercise open to scrutiny. Validity rests on review by peers. But this review cannot be just the preserve of a few in academia. Evidence from social research is widely used by professionals, administrators, policy-makers and the general public. Before use it is assessed. Peer review cannot be confined to a few academic colleagues where the subject is human transactions. Those researched and others sharing the evidence produced need to be able to act as peers and review from an informed position.

Social science is now big business. It contains the large numbers in higher education. It feeds the media. It is an important influence on policy and practice. We all use its ideas and evidence. We should also be active participants in assessing its quality. Peer review should not be left to a few experts, often colleagues of the researcher. For such a lay review to be possible there has to be sufficient information published. Research has to be genuinely public if those who use it are to be in a position to assess its credibility.

The crucial information is on the assumptions about the human condition built into the models that are used to produce hypotheses about social life and guide the choice of research methods. If humans are modelled as computers there will be a very different logic to enquiry than if they are seen as scientists. One will treat them as information-processors, the other as producers of that information. The former will proceed through setting them problems to solve, the latter will observe how they think through and test hunches for themselves. There are hundreds of metaphors and models of the human condition and hence hundreds of research methods. They change rapidly and vary country by country. Assessing limitations is not easy.

To illustrate the relation between methods, the models that justify their use and the way they have changed, here are three sketches of social research at the start, middle and end of the twentieth century.

First, come into the London School of Economics (LSE) shortly after Hobhouse had been appointed as the first Professor of Sociology in 1907. With two collaborators, one of whom, Ginsberg, succeeded him as Martin White Professor, he was throwing anthropological accounts and traveller’s tales into wastepaper baskets according to the stage of development each society was judged to have reached. Hobhouse pulled in evidence from any source into this evolutionary bucket model. The product of this particular effort was a statistical study, extracted from the baskets, comparing the cultures of ‘simpler people’ at different stages of their evolution (Hobhouse, Ginsberg and Wheeler 1915).

Now return to the LSE in the early 1950s. Floud, Halsey and Martin are researching into the chances of boys from different social classes entering grammar schools. The design is quasi-experimental. The boys are categorized accordin...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Introduction
• PART ONE Approaches to Social Research
• PART TWO Techniques for Collecting Information
• PART THREE Personal, Professional and Political Influences
• References
• Index