1
Data, Measurement, and Models
1.1Introduction
Journal of the American Statistical Association
Quintessentially, statistics is about solving problems; data (measurements or observations) relevant to these problems are collected, and statistical analyses are used to provide useful answers. But the path from data collection to analysis and interpretation is often not straightforward. Most real-life applications of statistical methodology have one or more nonstandard features, meaning in practice that there are few routine statistical questions, although there are questionable statistical routines. Many statistical pitfalls lie in wait for the unwary. Indeed, statistics is perhaps more open to misuse than most other subjects, particularly by the nonstatistician with access to powerful statistical software. The misleading average, the graph with âfiddled axes,â the inappropriate p-value, and the linear regression fitted to nonlinear data are just four examples of horror stories that are part of statistical folklore.
Statisticians often complain that many of those working in the behavioral sciences put undue faith in significance tests, use complex methods of analysis when the data merit only a relatively simple approach, and sometimes abuse the statistical techniques they are employing. Statisticians become upset (and perhaps feel a little insecure) when their advice to, say, âplot a few simple graphs,â is ignored in favor of a multivariate analysis of covariance or similar statistical extravagance.
However, if statisticians are at times horrified by the way in which behavioral scientists apply statistical techniques, behavioral scientists may be no less horrified by many statisticiansâ apparent lack of awareness of what stresses behavioral research can place on an investigator. A statistician may, for example, demand a balanced design with 30 subjects in each cell so as to achieve some appropriate power for the analysis. But it is not the statistician who is faced with the frustration caused by a last-minute phone call from a subject who cannot take part in an experiment that has taken several hours to arrange. Again, the statistician advising on a longitudinal study may call for more effort in carrying out follow-up interviews so that the study avoids statistical problems produced by the presence of missing data. It is, however, the behavioral researcher who must continue to persuade people to talk about potentially distressing aspects of their lives, who must confront possibly dangerous respondents, or who arrives at a given (and often remote) address to conduct an interview, only to find that the person is not at home. Many statisticians often do not appear to appreciate the complex stories behind each data point in many behavioral studies. One way of improving the possible communication problems between behavioral scientist and statistician is for each to learn more about the language of the other. There is already available a plethora of, for example, âStatistics for Psychologistsâ books, but sadly, (as far as we know) no âPsychology for Statisticiansâ equivalent. Perhaps there should be?
Having outlined briefly a few caveats about the possible misuse of statistics and the equally possible conflict between statistician and behavioral scientist, it is time to move on to consider some of the basics of behavioral science studies and their implications for statistical analysis.
1.2Types of Study
It is said that, when Gertrude Stein lay dying, she roused briefly and asked her assembled friends, âWell, whatâs the answer?â They remained uncomfortably quiet, at which she sighed, âIn that case, whatâs the question?â
Research in the behavioral science, as in science in general, is about searching for the answers to particular questions of interest. Do politicians have higher IQs than university lecturers? Do men have faster reaction times than women? Should phobic patients be treated by psychotherapy or by a behavioral treatment such as flooding? Do children who are abused have more problems later in life than children who are not abused? Do children of divorced parents suffer more marital breakdowns themselves than children from more stable family backgrounds?
In more general terms, scientific research involves a sequence of asking and answering questions about the nature of relationships among variables (e.g., How does A affect B? Do A and B vary together? Is A significantly different from B? and so on). Scientific research is carried out at many levels that differ in the types of question asked and therefore in the procedures used to answer them. Thus, the choice of which methods to use in research is largely determined by the kinds of questions that are asked.
Of the many types of investigation used in behavioral research, the most common are perhaps the following:
â˘Surveys
â˘Experiments
â˘Observational studies
â˘Quasi-experiments
Some brief comments about each of these four types are given below; a more detailed account is available in the papers by Stretch, Raulin, and Graziano, and by Dane, all of which appear in the second volume of the excellent Companion Encyclopedia of Psychology (see Colman, 1994).
1.2.1Surveys
Survey methods are based on the simple discovery that âasking questions is a remarkably efficient way to obtain information from and about peopleâ (Schuman and Kalton, 1985, p. 635). Surveys involve an exchange of information between researcher and respondent; the researcher identifies topics of interest, and the respondent provides knowledge or opinion about these topics. Depending upon the length and content of the survey as well as the facilities available, this exchange can be accomplished via written questionnaires, in-person interviews, or telephone conversations; and, in the 21st century, surveys via the Internet are increasingly common.
Surveys conducted by behavioral scientists are usually designed to elicit information about the respondentsâ opinions, beliefs, attitudes, and values. Perhaps one of the most famous surveys of the 20th century was that conducted by Alfred Charles Kinsey, a student of human sexual behavior in the 1940s and 1950s. The first Kinsey report, Sexual Behavior in the Human Male, appeared in 1948 (see Kinsey et al., 1948), and the second, Sexual Behavior in the Human Female, in 1953 (see Kinsey et al., 1953). It is no exaggeration to say that both reports caused a sensation, and the first quickly became a bestseller.
Surveys are often a flexible and powerful approach to gathering information of interest, but careful consideration needs to be given to several aspects of the survey if the information is to be accurate, particularly when dealing with a sensitive topic. Having a representative sample, having a large-enough sample, minimizing nonresponse, and ensuring that the questions asked elicit accurate responses are just a few of the issues that the researcher thinking of carrying out a survey needs to consider. Readers are referred to Bradburn et al. (2004) and Tourangeau et al. (2000) for a broad coverage of practical advice for questionnaire construction and Laaksonen (2018), Groves et al. (2009), de Lee...