A Student's Guide to Analysis of Variance

Maxwell Roberts, Riccardo Russo

• Publisher: Routledge
• Year: 2014
• eBook ISBN:
  9781317725053
• Topic: Ciencias sociales
• Subtopic: Probabilidad y estadística

In the investigation of human behaviour, statistical techniques are employed widely in the social sciences. Whilst introductory statistics courses cover essential techniques, the complexities of behaviour demand that more flexible and comprehensive methods are also employed. Analysis of Variance (ANOVA) has become one of the most common of these and it is therefore essential for both student and researcher to have a thorough understanding of it.
A Student's Guide to Analysis of Variance covers a range of statistical techniques associated with ANOVA, including single and multiple factor designs, various follow-up procedures such as post-hoc tests, and how to make sense of interactions. Suggestions on the best use of techniques and advice on how to avoid the pitfalls are included, along with guidelines on the writing of formal reports.
Introductory level topics such as standard deviation, standard error and t-tests are revised, making this book an invaluable aid to all students for whom ANOVA is a compulsory topic. It will also serve as a useful refresher for the more advanced student and practising researcher.

Chapter 1

---

Introduction

---

A book devoted to a single statistical test may seem excessive, but Analysis of Variance is more a collection of data analysis techniques than a single statistical test. In trained hands, they are extremely powerful and can make sense of the most convoluted data sets, but in untrained hands they can occasionally be dangerous. However, we will be starting at the basics and working forwards slowly. Although formulae and worked examples are given, we will keep the numbers of subjects and the scores low, simplifying the worked examples. In the first two chapters, we will begin by revising the most important topics that you were taught on your introductory courses. We will then discuss experimental designs in which a single independent variable can have three or more levels, and then designs with more than one independent variable. Hence, if you have ever been frustrated by being restricted to trying studies with one independent variable and two levels, or have been forced to choose between two alternative independent variables when you would have preferred to try both, then you will find the scope of these more advanced statistical tests to be liberating. However, first we must start with the basics, and to begin with we will recap on some of the key phrases that the remainder of the book will assume you are familiar with.

Experiment

For an experiment, at least two groups of subjects, or the same group of subjects on at least two different occasions, are treated exactly alike in all important ways with one exception – the experimental treatment. Any differences observed in the behaviour between the conditions must have been caused by the difference in the experimental treatments. Typically, the experimental process has five components:

(a) Subjects are allocated randomly to different experimental treatments.

(b) The experimenter is careful to vary only the treatment of interest.

(c) The experimenter measures some aspect of the behaviour of the subjects.

(d) If there are differences between the groups, the experimenter concludes that these differences were caused by the treatment.

(e) The experimenter interprets these results.

Experiments have several advantages over other forms of research. In particular, they are the only way of showing that there is a direct causal relationship between a particular treatment and measured behaviour. As long as the experiment is performed properly, differences must be due to the treatment and nothing else. However, this is not the end of the story. The results may be open to interpretation, they may not generalise to other situations, or they may need to be replicated.

Sometimes you may run a study which appears to be an experiment but is not. For example, males and females are often compared. These studies are not experiments; they are quasi-experiments. The experimenter is not controlling the classification – it is not possible to decide randomly whether a subject will be male or female – and this means that the exact cause of any difference can never be known.

Variables

A variable has the property that it can take different values. In other words, it varies. All psychologists are looking for relationships between variables. In experiments there are two types of variable. The independent variable is manipulated by the experimenter. This is also known as the treatment variable. Independent variables have at least two different levels; these are the experimental conditions. The dependent variable is measured by the experimenter in order to investigate the effects of manipulating the independent variable, hence it depends upon the independent variable for its value. Good dependent variables are readily observable, easily expressed as numbers, and measure the behaviour that they are intended to measure (i.e. they are valid). In quasi-experiments, instead of independent variables there are classification variables. Subjects are still assigned to groups, but the assignment is out of the control of the experimenter.

Samples and populations

Population refers to everyone. Thus you could talk about the population of UK university students, in which case you would be referring to every current university student in the country. Psychologists like to draw conclusions that apply to everyone, but because they do not have time to test everyone, they test samples from populations instead. They then try to decide whether the results they found for the samples apply to the entire populations from which they were drawn.

Experimental design

For a between-subjects design, also called an independent subjects design, two or more separate, completely independent groups of subjects are tested. Each group is assigned to a different experimental condition. Each subject contributes one single score to the final analysis. These designs have the advantage that you do not need to worry about fatigue or practice effects due to subjects' having to perform lots of different tasks. They have the disadvantage that they are less powerful than the alternative, so that many more subjects are required (see below). It is also possible that, due to bad luck, groups may be assembled that differ even before you test them. In order to avoid this problem, plenty of subjects need to be run, and they must be allocated to groups randomly or as unsystematically as possible, so that it is unlikely that groups are being created that differ.

For a within-subjects design, sometimes called a repeated measures design, one single group of subjects is tested once with each of the experimental conditions, and there must be at least two of these. Hence, each subject contributes at least two scores to the final analysis. Alternatively, similar subjects such as identical twins can be given one task each, but are then analysed as matched pairs using a within-subjects design. Within-subjects designs have the advantage that they are more powerful than between-subjects designs and so they need relatively fewer subjects. The disadvantage of this design is that steps must be taken to eliminate practice effects and fatigue effects. These are collectively known as order effects and can sometimes lead to uninterpretable results. The order of testing of conditions should be determined randomly to try to cancel these out. Alternatively, you can counterbalance, in which case exactly half of the subjects receive one order, and the other half receive the opposite order of conditions – more elaborate procedures are necessary if there are more than two. Order effects can occasionally cause very serious problems, so if you have the time and resources available to you, think about using a between-subjects design; sometimes you will not have a choice.

Error and bias

Whenever something is to be measured, there is always the potential for measuring it inaccurately. Some measurement errors can be reduced, but some can never be eliminated. As long as the size of the error is sufficiently small in relation to the size of the effect that you are trying to detect, measurement error will not be a problem. There are arguments against over-controlling an experiment in order to reduce errors to a minimum. The ‘perfect experiment’ would be impossible to implement, and if you can still get clear cut results despite ‘noisy data’ then this indicates that the results are robust.

Random errors are caused by non-systematic variations in performance in addition to those that are intended by the experimental manipulation. Hence, performance always differs from person to person, and even among individuals from trial to trial. People differ in motivation, knowledge, experience, health, attention span, etc. It is impossible to control for everything, although a within-subjects design can reduce these problems. Sometimes there may be random errors due to the actual running of the experiment, though these should not be too much of a problem unless the experimenter is very sloppy indeed. Random errors can never be completely removed, but if you run enough subjects, their effects should cancel each other out. Their consequences are to muddy results and make them less clear cut.

Systematic errors bias the results because they vary in tandem with the experimental conditions and confound them. This could be caused by, for example, testing subjects for one condition in the morning and for another in the afternoon. Unless you are careless, these errors should not be a problem.

In addition, floor effects and ceiling effects can bias the results of an experiment. A floor effect occurs when most subjects are performing so badly that their performance cannot get any worse. A ceiling effect occurs when performance is so good that it cannot get any better.

Statistical hypotheses: the null and the alternative hypothesis

Whenever a statistical procedure is used, at least one statistical hypothesis is being tested. This is distinct from a research hypothesis, which is a general statement that makes a prediction about the outcome of an experiment. A research hypothesis might be the prediction that one group will be, say, faster than another. A statistical hypothesis is amore explicit statement of the different possible outcomes that are associated with a particular research hypothesis. Instructions on how to write up a laboratory report often advise you to end the introduction section with your experimental hypotheses. You should take this to mean research hypotheses rather than statistical hypotheses. This is a common source of confusion.

Unfortunately, it is never possible to prove that anything is true. If you see 1,000 white swans, this does not prove that swans are always white. However, if you see just one black swan, this disproves that swans are always white. Because of this problem, statisticians think in terms of disproving that there is no difference between the means of two groups rather than proving that there is a difference between them. Disproving that there is no difference involves testing the null hypothesis.

For a two-condition experiment, the null hypothesis (H₀) would be that there is no difference between the means of the two conditions. The alternative hypothesis (H₁) would then be that there is a difference between the means of the two conditions. The purpose of a statistical test is to tell whether you should either reject the null hypothesis or fail to reject the null hypothesis. If you reject the null hypothesis, then you have disproved that the means of the groups do not differ, and you can safely accept the alternative hypothesis: that the means of the groups do differ. If the means are different, and this is an experiment, then the only possible reason for this difference is the experimental treatment, so you conclude that the different levels of the independent variable caused the behaviour of the groups to differ. If you fail to reject the null hypothesis, then you have failed to disprove that the means of the two groups do not differ, and you therefore cannot accept the alternative hypothesis. This does not prove that there is no difference between the means of the two groups. Instead this shows that there is not sufficient evidence to reject the null hypothesis. A new, better designed experiment might supply this evidence.

Statistical errors

Statistical testing is based upon confidence and not certainty. You could be extremely confident that the means of two groups differ, or you could lack confidence that the means of two groups differ. Without certainty, however, there is always the possibility that there might be a statistical error. A Type I Error occurs when the null hypothesis is rejected by mistake. You conclude that there is a difference between the means of two groups when in fact the null hypothesis is true for the population from which the sample was taken. In other words, you have concluded that the independent variable influences performance, when really it is not related to performance in the general population. A Type II Error occurs when there is a failure to reject the null hypothesis by mistake. You conclude that there is no evidence for a difference between the means of the two groups when, in fact, the null hypothesis is false for the population from which the sample was taken. In other words, you have concluded that there is no evidence that the independent variable influences performance, when really it does influence performance in the general population.

Significance levels

It is possible to calculate the probability that the difference between a pair of means arose due to chance. If the probability is unlikely enough, the null hypothesis is rejected and therefore the independent variable must have been responsible for the difference. Psychologists use an arbitrary cut-off point in order to decide whether to reject or fail to reject the null hypothesis. They thus decide upon a significance level. If the probability that the results arose due to chance is 0.05 (or 1/20, or 5%) or less, then you can say that p < 0.05 and the null hypothesis can be rejected. Thus, the difference in means between the two groups is said to be significant, as opposed to non-significant, never insignificant. Hence, statisticians are accepting that 1 time in 20, a Type I Error will be made. There is nothing you can do about this except replicate findings, although if you are really unlucky the replication may also be a Type I Error. Sometimes, a more stringent cut-off is used (p < 0.01). However, this increases the risk of making a Type II Er...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• List of figures
• List of tables
• List of boxes
• Preface
• Acknowledgements
• 1 Introduction
• 2 Averages, measures of dispersal and the t-test
• 3 Using variance to test hypotheses
• 4 Calculating F ratios for one-factor between-subjects designs
• 5 One-factor between-subjects ANOVA: advanced topics
• 6 Following up a one-factor between-subjects ANOVA
• 7 Calculating F ratios for one-factor within-subjects designs
• 8 An introduction to factorial designs and interactions
• 9 Calculating F ratios for two-factor between-subjects designs
• 10 Following up a two-factor between-subjects ANOVA
• 11 Interpreting two-factor mixed and within-subjects designs
• 12 Interpreting a three-factor ANOVA
• 13 Summary and frequently asked questions
• Appendix A: Writing up the results of Analysis of Variance
• Appendix B: Statistical tables
• Notes
• References
• Index