Statistics for Human Service Evaluation

Reginald O. York

• Publisher: SAGE Publications, Inc
• Year: 2016
• ISBN:
  9781483386713
• Edition: 1
• Topic: Social Sciences
• Subtopic: Sociology
• Index: Social Sciences

This practical book shows how both Excel® and SPSS® can be used for analyzing data for human service evaluation. Assuming no prior instruction for statistics, the text utilizes a "learn by doing" approach: readers see the use of statistics demonstrated and then are encouraged to apply their own data to statistical analysis with step-by-step guidance. Decision trees, practice exercises, and quizzes ensure readers will be well prepared to practice data analysis in a wide variety of human services situations.

1 Why Do We Use Statistics?Basic Concepts We Need to Know

Jamie has been collecting data on client outcomes for her Employment Support Program, which is designed to improve the chances of employment for a high-risk population. She compared clients who had completed the program with those on the waiting list for service, looking at whether the client had obtained employment. A year ago, she found that 60% of her 10 clients had obtained employment as compared to 40% of the 10 clients on the waiting list. Recently, she did another study and found that the 9 clients who had completed service did not have a higher rate of employment than those on the waiting list. She then compared her results with those of Parker, who runs a different employment program. He found a year ago that 65% of his 45 clients had obtained employment as compared to 45% of those on the waiting list.

Do you want to guess how well Parker’s clients are doing now? If his clients are still doing better than people who have not participated in his program, then why might his results have stayed the same while Jamie’s seem to have changed? Is there a difference between the effectiveness of their work over the last year?

You may have noticed that Parker had a much bigger number of clients in his study. Therefore, he can be more confident that his recent data will show a better outcome for his clients than those on the waiting list. Because Jamie’s sample sizes are smaller, her data are more likely to fail to be statistically significant. In other words, her initial results could too easily be explained by chance rather than something we can depend on to make predictions. If you have a large number of clients rather than a small number of clients and you find a modest difference that favors the clients, then when you conduct another study later, you are more likely to find your client group still has a better outcome than nonclients. Your results are more likely to be consistent because you have to a greater degree ruled out chance as the explanation for your data.

Why We Use Statistics

Statistics refers to the collection and analysis of data in a way that helps us make decisions. Should we continue the Support Group for Victims of Abuse? What about Jamie’s Employment Support Program? To answer these questions, we need to collect data on client growth and draw conclusions about the future effectiveness of the program. We might find that it is a great success, which means we should tell others about it. We might also find that it is a modest success or no success at all.

Where can statistics help us in this endeavor? We might find that a small number of clients had a small amount of growth. If so, can we be confident that a repeat of this program would bring modest success to another small number of clients? It would probably not be wise to bet our money on repeat success in this situation. Why? Because a small amount of growth experienced by a small number of people is not likely to be statistically significant. This means that we cannot adequately say that our treatment, rather than chance, is the explanation of our data.

Let’s examine the issue of chance. Suppose someone came into a room with a number of people and announced that he could tell whether or not someone was right-handed or left-handed by looking intensely into their eyes. Suppose further that someone else in the room said “Show me,” to which this person looked into her eyes and said, “You are right-handed.” Would you be a believer in this person’s ability to use intense eye inspection to detect whether someone was right-handed or left-handed? I would think not. Why? Because only about 10% to 15% of people are left-handed, so someone who guesses that another person is right-handed has the odds on his or her side. In other words, such a person could be right just by chance.

Suppose that you are examining a counseling program for at-risk youth at your middle school. You have been providing counseling to help students improve their confidence in the belief that they will someday graduate from high school. You have been collecting data using a special scale that measures confidence in continuing in school. To examine the effectiveness of this program, you have collected data at the beginning and end of the counseling. Suppose that the mean (average) score for confidence is 21.2 at the beginning of the program and the mean score after 3 months is 25.6. Given that higher scores on this scale mean more confidence, you have some evidence that your program is effective, especially if you can adequately rule out chance as the explanation for the increase in mean score. You rule out chance as a good explanation by subjecting your data to statistical analysis using a statistical test.

Now, what if you measured a single client on anxiety once a week during a period before treatment and did the same for the client during the treatment period? Suppose you found the following scores on your anxiety scale (where higher scores mean more anxiety): 24, 25, 21, 19, 18. These scores were going down before treatment began, indicating that the client’s anxiety was lessening. Suppose further that the weekly anxiety scores during the treatment period were 18, 19, 17, 17, and 16. The mean of these treatment scores is 17.4, while the mean of the previous scores is 21.4. This might seem to indicate that your treatment was a success, but don’t forget that the scores were going down before treatment. If we were to project the baseline scores (taken before treatment) into the treatment period, we would see a pattern that looks a lot like the treatment scores, suggesting that the treatment did not make a difference. To demonstrate otherwise, we would need to see a pattern of treatment scores that was distinctly different from the projected trend. In this situation, perhaps we would conclude either that treatment was not needed (the client was already improving) or that a different target behavior should be the focus of the service provided.

When we evaluate human services, we could make a mistake: We could conclude that our treatment has been effective because our clients have had a measured improvement in scores, when, in fact, we should conclude that our data can be easily explained by chance. In this event, we should not be confident that applying this treatment with other clients will bring about similar success. In other words, before continuing to use the treatment, we should use statistics to test whether chance is the best explanation of our data.

And let us not forget the issue of accountability in the administration of human services. Increasingly, those who hold the “purse strings” expect data to be used to justify the continued expenditure of funds. Statistics will naturally be a big part of this task.

What You Will Find in the Rest of This Chapter

This chapter will give a preview of what this book is about and discuss some basic concepts that guide the statistical analysis of data. While the focus is upon the analysis of data for evaluative studies, the information in this book will also be useful for descriptive and explanatory studies in which you are, respectively, describing your clients or examining the relationships between client characteristics (e.g., Do older clients experience more growth?). In addition, statistics useful for evaluation of human services are useful for studies in other fields. However, the primary focus here is on evaluation research, in which you evaluate the outcomes of an intervention, and the selection of statistics for examination is guided by this interest.

There is a quiz at the end of this chapter which you can use for two purposes. First, you could take it right now to see whether you are already familiar with the concepts in this chapter. Second, you could test yourself after you have read the chapter to see whether you need further review. There is also a chapter glossary, which you can use to review and test your understanding of key terms.

This book is designed to be an extremely user-friendly approach to using statistics for answering research questions. The goal is that you will learn how to use statistics by engaging in concrete tasks related to data of interest to you. If you employ the statistical concepts and methods demonstrated in these pages to conduct studies, you will learn about the essentials of statistics in a way that will likely stay with you after this learning expedition is done.

Illustrated in this chapter is a process for data analysis:

1. What is our research question?
2. What is the structure of our data?
3. What statistical test will we use?
4. How do we use the computer for statistical analysis?
5. What are our conclusions?

This process will be illustrated in this chapter and should reveal the user-friendly approach to statistics that is employed throughout this book. But first, let’s examine some basic concepts to get familiar with the nature of statistics.

Two Key Issues Addressed by Data Analysis—Practical Significance and Statistical Significance

The statistical examination of data addresses two fundamental issues:

1. Are my data clinically noteworthy?
2. Are my data easily explained by chance?

The first question refers to practical significance while the second one refers to statistical significance. We have discussed the basic concept of statistical significance in the first section of this chapter. Now we will expand on that discussion of statistical significance and add the theme of practical significance.

Statistical Significance

As previously illustrated, a major issue in regard to statistical significance is the extent to which your data can be explained by chance. How much can you depend on your data to be telling you about something real, rather than a phenomenon that occurred just by chance?

The statistical tests you’ll learn about in this book will yield a value of p. The value of p can range from 0 to 1.0. A value of 0 means that chance does not explain your data at all, while a value of 1.0 means that chance explains your data completely. In other words, a p value of 1.0 means all you have is chance, not data that can be treated seriously. When you do studies and when you read other people’s research, you will not likely see p values of 0 or 1.0. Indeed, these values are so rare that you should question such a result if it pops up on your computer. Instead, you will typically see values like .23 or .67 or .03 and so forth.

You have probably seen the expression “p < .05” in research reports. This statement means that these data would be expected to occur by chance fewer than 5 times in 100, and such data are normally held to have met the standard for statistical significance in the social sciences. So, if you employ this standard in your research, you will be hoping for a p value of less than .05 in order to say that your data support your expectations (i.e., your hypothesis).

Your hypothesis is your educated guess about what you will find when you analyze your data. You might state your hypothesis as “Posttest scores for depression will be lower than pretest scores.” You would then collect pretest and posttest scores and subject these data to statistical analysis. If the posttest scores were lower and the difference between pretest and posttest scores were statistically significant, you could say your data supported your hypothesis.

The standard of .05 for stating that data support a hypothesis has been advocated by those who write about research and statistics and has been widely accepted. However, a decision to use the standard of .10 (10 times in 100) instead of .05 would be just as scientifically valid because the standard is a matter of judgment. When making this judgment, we should consider the relative advantage of being more or less conservative in the interpretation of our data. If we are engaging in evaluative research and wish to make sure we do not conclude that an ineffective treatment is effective, we will be more conservative and employ a low value of p as our standard. We might use the standard of .05 or an even more conservative standard of .01. On the other hand, if we want to make sure we do not erroneously conclude that an effective treatment is not effective, we would choose a less conservative standard, like .10. Be aware, however, that the further you depart from the standard of .05, the less likely you will find agreement among those who read your reports.

As indicated above, the standard you select should be guided by the kind of risk that is more important, given what you are studying. In research about drugs, you are likely to find a conservative standard, maybe .01 or even .001. That is because of the danger inherent in a false positive (i.e., believing an ineffective drug is actually effective).

If you are interested in what determines statistical significance, read Insight Box 1.1, where sample size, magnitude of data, and variance are discussed.

Insight Box 1.1: What Determines the Level of Statistical Significance?

There are three things that determine the level of statistical significance for your data—sample size, magnitude of the data, and variance. Sample size is the number of people you have collected data on (or the number of pieces of data). Magnitude deals with the question of how much change has occurred. Variance measures the extent that the scores on your key variables are either similar to each other or different from eac...