Although it might seem a bit strange to discuss research design in a book on IBM SPSS Statistics, it is an essential part of the research process that has implications for the quality of the data collected and analysed. The data you enter must come from somewhereāresponses to a questionnaire, information collected from interviews, coded observations of behaviour or objective measurements of output or performance. The data are only as good as the instrument that you used to collect them and the research framework that guided their collection.
In this chapter various aspects of the research process are discussed that have an impact on the potential quality of the data. First, the overall design of the study is considered; this is followed by a discussion of some of the issues to consider when choosing scales and measures; and finally, some guidelines for preparing a questionnaire are presented.
Planning the Study
Good research depends on the careful planning and execution of the study. There are many excellent books written on the topic of research design to help you with this processāfrom a review of the literature to formulation of hypotheses, choice of study design, selection and allocation of participants, recording of observations and collection of data. Decisions made at each of these stages can affect the quality of the data you have to analyse and the way you address your research questions. In designing your own study, I would recommend that you take your time working through the design process to make it the best study that you can produce. Reading a variety of texts on the topic will help. A few good, easy-to-follow titles are listed in the Recommended Reading section at the back of the book.
To get you started, consider these tips when designing your study:
- ā¢Consider what type of research design (e.g. experiment, survey, observation) is the best way to address your research question. There are advantages and disadvantages to all types of research approaches; choose the most appropriate approach for your particular research question. Have a good understanding of the research that has already been conducted in your topic area.
- ā¢ If you choose to use an experiment, decide whether a between-groups design (different cases in each experimental condition) or a repeated measures design (same cases tested under all conditions) is the more appropriate for your research question. There are advantages and disadvantages to each approach, so weigh up each approach carefully.
- ā¢ In experimental studies, make sure you include enough levels in your independent variable. Using only two levels (or groups) means fewer participants are required, but it limits the conclusions that you can draw. Is a control group necessary or desirable? Will the lack of control group limit the conclusions that you can draw?
- ā¢Always select more participants than you need, particularly if you are using a sample of humans. People are notoriously unreliableāthey don't turn up when they are supposed to, and they get sick, drop out and don't fill out questionnaires properly! So plan accordingly. Err on the side of pessimism rather than optimism.
- ā¢ In experimental studies, check that you have enough participants in each of your groups (and try to keep them equal when possible). With small groups, it is difficult to detect statistically significant differences between groups (an issue of power, discussed in the introduction to Part Five). There are calculations you can perform to determine the sample size that you need. See, e.g. Stangor (2006).
- ā¢ Wherever possible, randomly assign participants to each of your experimental conditions, rather than using existing groups. This reduces the problem associated with non-equivalent groups in between-groups designs. Also worth considering is taking additional measurements of the groups to ensure that they don't differ substantially from one another. You may be able to statistically control for differences that you identify (e.g. using analysis of covariance).
- x27A2; Choose appropriate dependent variables that are valid and reliable (see discussion on this point later in this chapter). It is a good idea to include a variety of measuresāsome measures are more sensitive than others. Don't put all your eggs in one basket.
- ā¢ Try to anticipate the possible influence of extraneous or confounding variables. These are variables that could provide an alternative explanation for your results. Sometimes, they are hard to spot when you are immersed in designing the study yourself. Always have someone else (e.g. supervisor, fellow researcher) check over your design before conducting the study. Do whatever you can to control for these potential confounding variables. Knowing your topic area well can also help you identify possible confounding variables. If there are additional variables that you cannot control, can you measure them? By measuring them, you may be able to control for them statistically (e.g. using analysis of covariance).
- ā¢ If you are distributing a survey, pilot-test it first to ensure that the instructions, questions and scale items are clear. Wherever possible, pilot-test on the same types of people who will be used in the main study (e.g. adolescents, unemployed youth, prison inmates). You need to ensure that your respondents can understand the survey or questionnaire items and respond appropriately. Pilottesting should also pick up any questions or items that may offend potential respondents.
- ā¢ If you are conducting an experiment, it is a good idea to have a full dress rehearsal and to pilot-test both the experimental manipulation and the dependent measures you intend to use. If you are using equipment, make sure it works properly. If you are using different experimenters or interviewers, make sure they are properly trained and know what to do. If different observers are required to rate behaviours, make sure they know how to appropriately code what they see. Have a practice run and check for inter-rater reliability (i.e. how consistent scores are from different raters). Pilot-testing of the procedures and measures helps you identify anything that might go wrong on the day and any additional contaminating factors that might influence the results. Some of these you may not be able to predict (e.g. workers doing noisy construction work just outside the lab's window), but try to control those factors that you can.
Choosing Appropriate Scales and Measures
There are many different ways of collecting data, depending on the nature of your research. This might involve measuring output or performance on some objective criteria, or rating behaviour according to a set of specified criteria. It might also involve the use of scales that have been designed to operationalise some underlying construct or attribute that is not directly measurable (e.g. self-esteem). There are many thousands of validated scales that can be used in research. Finding the right one for your purpose is sometimes difficult. A thorough review of the literature in your topic area is the first place to start. What measures have been used by other researchers in the area? Sometimes, the actual items that make up the scales are included in the appendix to a journal article; otherwise, you may need to trace back to the original article describing the design and validation of the scale you are interested in. Some scales have been copyrighted, meaning that to use them you need to purchase official copies from the publisher. Other scales, which have been published in their entirety in journal articles, are considered to be 'in the public domain', meaning that they can be used by researchers without charge. It is very important, however, to properly acknowledge each of the scales you use, giving full reference details.
In choosing appropriate scales there are two characteristics that you need to be aware of:
reliability and validity Both of these factors can influence the quality of the data you obtain. When reviewing possible scales to use, you should collect information on the reliability and validity of each of the scales. You need this information for the Method section of your research report. No matter how good the reports are concerning the reliability and validity of your scales, it is important to pilot-test them with your intended sample. Sometimes, scales are reliable with some groups (e.g. adults with an English-speaking background) but are totally unreliable when used with other groups (e.g. children from non-English-speaking backgrounds).
Reliability
The reliability of a scale indicates how free it is from random error. Two frequently used indicators of a scale's reliability are
test-retest reliability (also referred to as 'temporal stability') and
internal consistency The test-retest reliability of a scale is assessed by administering it to the same people on two different occasions and calculating the
correlation between the two scores obtained. High test-retest correlations indicate a more reliable scale. You need to take into account the nature of the construct that the scale is measuring when considering this type of reliability. A scale designed to measure current mood states is not likely to remain stable over a period of a few weeks. The test-retest reliability of a mood scale, therefore, is likely to be low. You would, however, hope that measures of stable personality characteristics would stay much the same, showing quite high test-retest correlations.
The second aspect of reliability that can be assessed is internal consistency. This is the degree to which the items that make up the scale are all measuring the same underlying attribute (i.e. the extent to which the items 'hang together'). Internal consistency can be measured in several different ways. The most commonly used statistic is Cronbach's coefficient alpha (available using IBM SPSS Statistics; see
Chapter 9). This statistic provides an indication of the average correlation among all of the items that make up the scale. Values range from 0 to 1, with higher values indicating greater reliability.
While different levels of reliability are required, depending on the nature and purpose of the scale, Nunnally (1978) recommends a minimum level of .7. Cronbach alpha values are dependent on the number of items in the scale. When there are a small number of items in the scale (fewer than 10), Cronbach alpha values can be quite small. In this situation it may be better to calculate and report the mean interitem correlation for the items. Optimal mean inter-item correlation values range from .2 to .4 (as recommended by Briggs & Cheek 1986).
Validity
The validity of a scale refers to the degree to which it measures what it is supposed to measure. Unfortunately, t...
