Small Sample Size Solutions

A Guide for Applied Researchers and Practitioners

Rens van de Schoot, Milica Miočević, Rens van de Schoot, Milica Miočević

• Publisher: Routledge
• Year: 2020
• eBook ISBN: 9781000761085
• Edition: 1
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

Researchers often have difficulties collecting enough data to test their hypotheses, either because target groups are small or hard to access, or because data collection entails prohibitive costs. Such obstacles may result in data sets that are too small for the complexity of the statistical model needed to answer the research question. This uniquebook provides guidelines and tools for implementing solutions to issues that arise in small sample research. Each chapterillustrates statistical methods that allow researchers to apply the optimal statistical model for their research question when the sample is too small.

Thisessential book will enable social and behavioral science researchers to test their hypotheses even when the statistical model required for answering their research question is too complex for the sample sizes they can collect.The statistical models in the book rangefrom the estimation of a population mean to models with latent variables and nested observations, and solutions include both classical and Bayesian methods.All proposed solutions are described in steps researchers can implement with their own data and are accompanied with annotated syntax in R.

The methods described in this book will be useful for researchers across the social and behavioral sciences, ranging from medical sciences and epidemiology to psychology, marketing, and economics.

PART I

Bayesian solutions

1

Introduction to Bayesian Statistics

Milica Miočević

Department of Psychology, McGill University, Montreal, Canada

Roy Levy

T. Denny Sanford School of Social and Family Dynamics, Arizona State University, Arizona, United States of America

Rens van de Schoot

Department of Methodology and Statistics, Utrecht University, Utrecht, the Netherlands & Optentia Research Program, Faculty of Humanities, North-West University, Vanderbijlpark, South Africa

Introduction

Bayesian statistics are becoming more popular in many fields of science. See, for example, the systematic reviews published in various fields from educational science (König & Van de Schoot, 2017), epidemiology (Rietbergen, Debray, Klugkist, Janssen, & Moons, 2017), health technology (Spiegelhalter, Myles, Jones, & Abrams, 2000), medicine (Ashby, 2006), and psychology (Van de Schoot, Winter, Ryan, Zondervan-Zwijnenburg, & Depaoli, 2017) to psychotraumatology (Van de Schoot, Schalken, & Olff, 2017). Bayesian methods appeal to researchers who only have access to a relatively small number of participants because Bayesian statistics are not based on large samples (i.e., the central limit theorem) and hence may produce reasonable results even with small to moderate sample sizes. This is especially the case when background knowledge is available. In general, the more information a researcher can specify before seeing the data, the smaller the sample size required to obtain the same certainty compared to an analysis without specifying any prior knowledge.

In this chapter, we describe Bayes’ theorem, which is the foundation of Bayesian statistics. We proceed to discuss Bayesian estimation and Bayes Factors (BFs). The chapter concludes with a brief summary of take-home messages that will allow readers who are new to Bayesian statistics to follow subsequent chapters in this book that make use of Bayesian methods. The applications of Bayesian statistics described in this volume cover the following topics: the role of exchangeability between prior and data (Chapter 2, Miočević et al.), applying the WAMBS checklist (Chapter 3, Van de Schoot et al.) using informative priors when fitting complex statistical models to small samples (Chapter 4, Veen & Egberts), regression analysis with small sample sizes relative to the number of predictors (Chapter 5, Van Erp), data analysis with few observations from a single participant (Chapter 8, Lek & Arts), updating results participant by participant (Chapter 9, Klaassen), clinical trials with small sample sizes and informative priors based on findings from other trials (Chapter 10, Kavelaars), tests for evaluating whether a finding was replicated (Chapter 12, Zondervan-Zwijnenburg & Rijshouwer), and a comparison between frequentist two-step modeling and Bayesian methods with informative priors (Chapter 17, Smid & Rosseel). Due to space considerations, this chapter does not offer an exhaustive discussion of Bayesian statistics and the differences between Bayesian and classical (frequentist) statistics; for approachable texts on Bayesian statistics in the social sciences, we refer readers to books by Kaplan (2014) and Kruschke (2014), and the chapter by Gigerenzer (1993).

Bayes’ theorem

Bayesian statistics are a branch of statistics that implements Bayes’ theorem to update prior beliefs with new data:

$$p\left(\mathit{\theta }|data\right)=\frac{p\left(data|\mathit{\theta }\right)p\left(\mathit{\theta }\right)}{p\left(data\right)}\propto p\left(data|\mathit{\theta }\right)p\left(\mathit{\theta }\right)$$

(1.1)

where $\mathit{\theta }$ denotes a set of parameters (e.g., regression coefficients), $p(\mathit{\theta }|data)$ is the posterior distribution of the parameters, which was obtained by updating the prior distribution of the parameters, $p\left(\mathit{\theta }\right)$ , with the observed data represented by the likelihood function, $p(data|\mathit{\theta })$ . The term $p\left(data\right)$ is the marginal probability of the data that can be considered a normalizing constant that ensures that the posterior distribution integrates to 1. As the right-hand side of Equation 1.1 shows, excluding this term yields a result that is proportional to the posterior distribution.

In the Bayesian framework, the updated (posterior) beliefs about the parameters in a statistical model are used for inference. The posterior distribution can be summarized to report the probability that a parameter lies within a given range. Bayes’ theorem stems from the laws of conditional probabilities, which are not controversial. The controversial elements surrounding Bayesian statistics are whether to engage in Bayesian analysis and accept the requirement of specifying a prior distribution, and once the researcher chooses to use Bayesian inference, how to specify the prior distribution, $p\left(\mathit{\theta }\right)$ . Applied researchers are often advised to base their prior distributions on previous findings, meta-analyses, and/or expert opinion; for considerations related to the choice of source of prior information, see Chapter 2. The exact influence of the prior is often not well understood, and priors will have a larger impact on the results when sample size is small (see Chapter 3). Bayesian analyses of small data sets using priors chosen by the researcher can sometimes lead to worse estimates than those obtained using uninformative priors or classical methods (Smid, McNeish, Miočević, & Van de Schoot, 2019). Thus, priors should be chosen carefully.

To illustrate a Bayesian statistical analysis, consider a normally distributed variable $y$ (for example, IQ, used to illustrate Bayesian inference in the shiny application example from www.rensvandeschoot.com/fbi/; see also the Center for Open Science (OSF): https://osf.io/vg6bw/) with unknown mean $\mu$ and a known variance ${\sigma }^2$ . In the frequentist framework, one would collect a sample of data (IQ scores), $y_1,\dots y_n$ , compute the sample mean $\bar{y}$ , and use it as the estimate of the population mean of IQ. The standard error is a measure of the uncertainty surrounding the estimate.

In the Bayesian framework, the researcher would start the analysis by specifying a prior distribution for $\mu$ (population mean of IQ). When specifying a prior distribution, researchers have to select a distributional form (e.g., normal distribution, $t$ -distribution, beta distribution), and specify the parameters of the prior distribution, known as hyperparameters. A common choice of prior distribution for the population mean $\mu$ is th...

Table of contents

• Cover
• Half Title
• Series Page
• Title Page
• Copyright Page
• Table of Contents
• Introduction
• List of symbols
• PART I: Bayesian solutions
• PART II: n = 1
• PART III: Complex hypotheses and models
• Index