Practical Sampling

Gary T. Henry

• Publisher: SAGE Publications, Inc
• Year: 1990
• Print ISBN: 9780803929593
• eBook ISBN: 9781506320342
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

Sampling is fundamental to nearly every study in the social and policy sciences, yet clear, concise guidance for practitioners and graduate students has been difficult to find. Practical Sampling provides guidance for researchers dealing with the everyday problems of sampling. Using the practical design approach Henry integrates sampling into the overall research design and explains the interrelationships between research design and sampling choices. He lays out alternatives and implications of the choices using four detailed examples to illustrate the alternatives selected and the trade-offs made by applied researchers.

The author uses a narrative, conceptual approach throughout the book; mathematical presentations are limited to necessary formulas; and calculations are kept to the absolute minimum, making it an easily approachable book for any researcher, student or professional across the social sciences.

1

Introduction

Most data used in the social and policy sciences are collected from samples. Public opinion surveys, social experiments, and evaluations of educational innovations are examples of research where sampling is routinely used. In any research in which the findings are being extrapolated from the subjects or units under study to a larger study population, sampling is being utilized. Samples are so frequently utilized that counterexamples—studies where the entire study population is involved, such as the decennial U.S. census—are relatively rare. Without relying on sampling as the basis for collecting evaluative data, the risk and cost involved with adopting new methods of teaching or social service delivery would be difficult to justify. Evaluating the effectiveness of new programs would be prohibitive.

The introduction and use of probability sampling techniques have stimulated the growth of empirical social and policy research in the later 20th century. Despite the importance of sampling, researchers in the social and policy sciences receive little practice with sampling and often seek guidance concerning practial sampling applications. Assumptions about sampling methods often underlie analytical techniques that the researchers wish to utilize. Researchers often need to insure that the assumptions are justified.

The sampling dilemma is simple. Time and cost prohibit a researcher from collecting data on the entire group or population that is of interest for a particular study. However, researchers and the consumers of research are usually interested in the population rather than a subset of the population. Extending the study findings from a subset of the population, or “sample,” to the entire population is critically important to overcome the dilemma between cost and time on the one hand and information needs on the other. Knowledge of basic sampling logic and methods is fundamental to ascertain if study findings reasonably apply to the population represented by the study participants or the survey respondents.

Understanding sampling and its implications is especially important in policy research. Questions that commonly arise in conducting and using policy research directly related to understanding the choices made in sample design are: Is the target population for the policy or program defined in the same way as the population in the study? Have the methods for selecting the subjects or units biased the estimates needed by policymakers? Are estimates from the sample data precise enough for the study purpose? Table 1.1 highlights these three issues, the criteria used to make judgments about the usefulness of the study, and the potential implications for using the study results.

Using an example of an assessment of service needs for the advanced elderly, the impact of sampling design choices on the results can be shown:

• Population definition. A study is to provide a service needs assessment for all advanced elderly in a state. If the population from which the sample is drawn is current clients of publicly provided social and medical services, then elderly not being served but who need services are excluded. This choice would produce an underestimate of actual needs.
• Sampling methods. A sampling strategy that focuses on elderly living in group residences could bias the results. Group residences provide the highest level of care in many instances. Therefore, the estimate of needs, when extrapolated to the entire elderly population of the state, may be too high. The sample would not include enough self-sufficient elderly living on their own or with family members.
• Precision of the estimate. An estimate, say a mean or proportion, produced from a sample, even when the sampling method is unbiased, is subject to fluctuation. An uninformed consumer of the study results may assume that an estimate is exact and place too much credence in the “exact” estimate. For instance, an estimate from the needs assessment may indicate that 63.4% of the advanced elderly require service. For a small sample, the researcher may be reasonably confident the true mean is between 54% and 73%. Assuming 500,000 advanced elderly in the population, the estimate of the service population varies by nearly 100,000 from one end of the range to the other. In this case, the estimate may be too imprecise to establish policies and programs for service delivery tothe advanced elderly.

SAMPLE DEFINED

The word “sample” is used in a variety of ways in scientific and everyday language. For example, Mark and Workman (1987, p. 47) point out that, “To the chemist, the word sample conjures up the image of a small pile of something, or perhaps a small container of liquid, paste, or maybe ‘gunk,’ of which the composition must be determined.” The sample is intrinsically important to the chemist. It may be all of the substance that was available, for instance from a crime scene, or part of a larger mass that has been selected. But determining the composition of the sample is an end in itself. Discovery of arsenic in the tea is the forensic chemist’s charge, not a representation of the population of tea.

TABLE 1.1
Issues in Sample Design for Policy Research

---

Issue	Criterion	Implication
Population Definition	Consistency of target population and study population	Study population yields biased results by including members not in target population or leaving out members who are in target population
Sampling Method	Sample selection equally likely to select any member of study population	Sampling methods yield biased results if some study population members are more likely to be selected than others
Precision of Estimate	Estimate precise enough to inform policy decision	All samples yield estimates, not exact figures. Lack of precision can impact on the decisions to be made

---

The chemist’s sample can be classified as a specimen, where the particular case is important. In contrast, sample, as it is used in the research literature and in this book, means a subset of the population that is used to gain information about the entire population. A sample in this sense is a model of the population. A good sample will represent the population well. The sample does not have intrinsic interest to the social or policy scientist: it is a tool to find out about the population.

Two questions arise naturally from this discussion:

• How should one select a sample that will be used to represent the population?
• How do we judge whether the sample represents the population well?

Guidance concerning the first question will be addressed in the next chapter and continued throughout the book. In Chapter 3, possible sources of error in the sample and a framework for making choices in the sample design process are presented.

Before turning to these discussions, a comment on the use of the word “represent” is germane. A sample is used to represent the population. Thus, it is a model or representation of the population. Adding the term “representative” to “sample,” as in the commonly used phrase “representative sample,” provides no additional information about the sample. The adjective “representative” has no technical definition and simply represents a subjective judgment on the part of the term’s user. No objective criteria are established to determine if a sample is or is not representative. Frequently, however, “representative sample” is the only description of the sample that is provided. Dropping the adjective and including a description of the sample selection process and information on the correspondence between the sample and the population is recommended. The importance of the description is introduced in the next section.

SAMPLING AND VALIDITY

Rarely can a researcher collect data on all the subjects of interest in a particular study. Samples provide a practical and efficient means to collect data. The sample serves as a model of the population. However, for a researcher to extend study findings to the population, the model must be an accurate representation of the population.

The ability of a researcher or user of a study to extend findings beyond sample individuals, time, and place is referred to as “external validity” (Campbell & Stanley, 1963; Cook & Campbell, 1979). Cook and Campbell pose the central question for external validity by asking, “Given that there is probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times?” (1979, p. 39). For example, researchers find that using a computer-assisted instruction software package for reading in the third grade in an inner-city school improved the students’ vocabulary and comprehension. Consumers of this study could reasonably ask how the instructional method would work in rural schools? How about with fourth graders? Are the gains a product of the novelty of using the computer in the classroom that would not occur when the novelty wears off?

The ability to generalize study findings is a function of the sample from which the data are actually obtained. Both sampling design and execution have an impact on generalizability. The practical sampling approach taken in this book emphasizes both design and execution, for both can affect the validity or total error of the research. Sample design includes choosing an appropriate selection technique, such as random digit dialing, and determining the number of cases needed for the study. Executing a design includes obtaining a comprehensive listing of the population for the study, obtaining the data reliably, and insuring that responses are actually received from a group whose composition accurately represents the population. Any plan or action that affects the composition of the group from whom data are actually collected has a bearing on the generalization of the results. Therefore, practical sampling design must be integrated throughout research design and execution.

In addition to external validity, the sample design is directly affected by and directly affects two statistical validity considerations included in Cook and Campbell’s discussion of validity (1979). Statistical conclusion validity is the ability to reach conclusions about relationships that appear in the sample data, that is, covariation. Statistical tests are generally used to examine whether the relationship that is observed is due to change. Or as Kraemer and Thiemann state, “A statistical test defines a rule that, when applied to the data, determines whether the null hypothesis can be rejected, i.e., whether the evidence is convincing beyond reasonable doubt” (1987, p. 23). Because these tests are sensitive to both the size of the relationship (effect size) and the size of the sample, the sample size can be critical to avoid “false conclusions about covariation” (Cook & Campbell, 1979, p. 37).

Small sample size may contribute to a conservative bias (Type II error) in the application of a statistical test. A Type II error occurs when a null hypothesis is not rejected although in fact it is false. In this situation, the program or intervention being tested is judged ineffective even though it does have an effect. However, the “reasonable doubt” criterion may be impossible to meet given the expected effect size and the actual sample size. The conservative bias occurs in instances where a small effect or covariation is true but the sample size is not sufficient for the effect to register above the statistical significance threshold. This phenomenon can be especially frustrating in evaluations of pilot programs where the number of participants and sample size are small. Effects resulting from the program that are small but meaningful can fall prey to the lack of statistical significance. Therefore, evaluators may wrongly conclude that the program failed to work effectively. This problem of sample size and statistical conclusion validity, referred to as power, is discussed in Chapter 7. (See Kraemer & Thiemann, 1987; Oakes, 1986; or Lipsey, 1989; for a more detailed explanation.)

A second aspect of statistical conclusion validity—reliability of measures—impacts sampling considerations. The less reliable an instrument is, the greater the observed variation in the data (Cook & Campbell, 1979). When observed variation increases, it becomes more difficult to reject the null hypothesis, even though a true relationship exists. To some extent, larger sample sizes can compensate for the increased variation, assuming the instrument is unbiased. However, to compensate for the inflation of the variance due to the lack of reliability of the instrument, it must be recognized and accounted for early in the design process.

WHY SAMPLE?

Given the mine field of validity concerns, a researcher is likely to ask, “Why sample?” Sampling is ultimately a practical means to an end. Researchers usually begin with a target population, often defined by a policy or program about which they ask a question. For example, a researcher could ask “Do developmental preschool programs for at-risk 4-year-olds improve cognitive gains and decrease the need for special-education assistance for these students in later years?” A target population of at-risk 4-year-olds as defined by policymakers is included in the research question.

The researcher transforms the research question into a feasible empirical project through the use of sampling. Clearly, it is not likely that all at-risk 4-year-olds can be provided with the development program and tested over a period of years to determine the impact of the program. Resource limitations prevent this. In this case, the limitations occur from both the programmatic and research concerns. Finding funds, facilities, and trained personnel to provide the developmental programs would be difficult. Equally difficult would be the investment in data collection, analysis, and follow-up needed for the evaluation. Nor would it be prudent to expend public funds for the program without an evaluation of its impact.

Sampling allows the use of a subset of the target population for testing the program. The principal reason for sampling is the resource constraint on the research project. But sampling can also improve quality. For example, limitations on the number of trained individuals that can competently administer pretests on 3- and 4-year-olds may necessitate hiring untrained staff or utilizing tests for at-risk status that are too simplistic to produce reliable results. Sampling can allow resources to be directed to improve the amount and quality of data on each individual and minimize problems of missing data.

Researchers can encounter situations where sampling is not advisable. Two situations come to mind: sampling from small populations and sampling that may reduce credibility of results. When dealing with small populations (less than 50 members), collecting data on the entire population often improves the reliability and credibility of the data. The influence of a single extreme case or outlier in the data is much more pronounced with small samples, and testing hypotheses becomes much simpler with population data. Also, if study consumers know that a “unique case” was omitted from the sample, the credibility of the results can be damaged. This type of problem is more likely to occur with a small population where consumers have more detailed information concerning individual members of the population.

The credibility of a study may also be adversely influenced by sampling in a study that may lead to recommendations about the distribution of public funds. For example, using a sample of political subdivisions—cities and counties—to ...

Table of contents

• Cover page
• Title
• Copyright
• Contents
• Acknowledgments
• 1 Introduction
• 2 Sample Selection Approaches
• 3 Practical Sample Design
• 4 Four Practical Sample Designs
• 5 Sampling Frames
• 6 Sampling Techniques
• 7 Sample Size
• 8 Postsampling Choices
• References
• About the Author