Data Collection Research Methods in Applied Linguistics

Heath Rose, Jim McKinley, Jessica Briggs Baffoe-Djan

• Editorial: Bloomsbury Academic
• Año: 2019
• ISBN:
  9781350025851
• Edición: 1
• Categoría: Langues et linguistique
• Categoría: Linguistique

The successful collection of data is a key challenge to obtaining reliable and valid results in applied linguistics research. Data Collection Research Methods in Applied Linguistics investigates how research is conducted in the field, encompassing the challenges and obstacles applied linguists face in collecting good data. The book explores frequently used data collection techniques, including: * interviews and focus groups
* observations
* stimulated recall and think aloud protocols
* data elicitation tasks
* corpus methods
* questionnaires
* validated tests and measures Each chapter focuses on one type of data collection, outlining key concepts, threats to reliability and validity, procedures for good data collection, and implications for researchers. The chapters also include exemplary research projects, showcasing and explaining for readers how the technique was used to collect data in a successfully published study. This book is an essential resource for both novice and experienced applied linguists tackling data collection techniques for the first time.

1

An Introduction to Research Methods

Pre-reading activities

Think

Before reading this chapter, make two lists of all the research designs (e.g. experiments, case studies) and data collection methods (e.g. interviews, questionnaires) that you can think of. Consider the differences between a research design and a data collection method.

Discuss

In small groups, or as a class, compile your lists into a master list. Then attempt to further organize these terms into categories based on similarities and differences. Then discuss the following: (1) How difficult is it to organize methods/designs according to a qualitative and quantitative dichotomy? (2) Is there a better way to categorize these methods and designs?

Imagine

Imagine you are conducting a study of second language acquisition during a study abroad programme. Your participants are based in Turkey and are doing a 6-month study abroad in Ireland. You are interested in discovering the language proficiency gains in English in terms of reading, writing, speaking and listening proficiency, plus knowledge gains in vocabulary. Furthermore, you want to understand if learner motivation affects gains. Design a study to research this topic. What data collection instruments might be useful?

Introduction: Why study research methods?

An understanding of research methods is essential for all applied linguistics postgraduate students, novice researchers, as well as practising language teachers. A knowledge of research methods can help students to develop a critical eye when reading research. Research methods training is essential for postgraduate researchers embarking on research for the first time or entering into unfamiliar research domains. An understanding of research methods can also heighten novice researchers’ awareness of the repertoire of approaches to research design, data collection and data analysis available to them. An understanding of research methods is also essential for language teachers, who are increasingly encouraged to undertake practitioner-based evaluations of their teaching practices and curriculum innovations. While the aim of such research may not be formal publishing, knowledge of research methods can help inform teachers of the various ways to collect data within educational contexts. Research methods can make classroom-based inquiry more robust, thus giving the findings from such research more clout in terms of impact on practice. Finally, a good grounding in research methods can help people to be more discerning consumers of research in general.

Key terms

In this book, we make an important distinction between research design and data collection methods. While both are part of research methodology, a research design refers to the methodological structure of a study, which informs the research approach. Popular research designs include, but are not limited to, surveys, case studies, experiments, action research, field research, corpus research and ethnographies. Data collection refers to the actual methods used to gather data for analysis. Popular data collection research methods include, but are not limited to, questionnaires, interviews, focus groups, tests, language elicitation tasks, corpora and observations. Research designs do not dictate how data are collected but rather provide the framework and philosophy within which a researcher collects the data. For example, survey research, which involves the widespread examination of the prevalence of a particular construct within a population, is often paired with questionnaires as a data collection method but could equally make use of data collection techniques such as document analysis, depending on its appropriateness to the topic. Field research, which involves a researcher exploring in-context phenomena, could make use of a combination of data collection methods such as interviews, observations and questionnaires.

Book structure

While the focus of this book is on data collection in applied linguistics research, this opening chapter provides an overview of research designs in order to establish the methodological foundation upon which to discuss the collection of data. Many research methods books tend to focus on design as methodology, neglecting the procedures of collecting the actual data. To fill this gap, the focus of our book is on exploring research methods through the lens of data collection to provide a practical perspective of applied linguistics research. Of course, research designs are important in providing methodological structure to data collection; thus, we first focus on common research designs in this introductory chapter. For a fuller overview of methodology from a research design perspective, readers should consult the resource, Research Methods in Applied Linguistics (Paltridge & Phakiti, 2015).

Popular research designs

In this section, we explore research designs that are prevalent in applied linguistics research. Rather than present these designs in the abstract, we do so via an illustrative example of a hypothetical study. This hypothetical study centres around a language education researcher who wants to conduct research into the effects of a new language learning software called Software X on students’ learning.

Experimental and quasi-experimental research

Experiments are considered the gold standard of showing causation in research methodology. Indeed, the rigour of experimental designs is often used as a benchmark to evaluate the validity and reliability of other methods. A true experiment attempts to isolate cause and effect of a ‘thing’ being manipulated (referred to as the independent variable) and the effects of that manipulation on the ‘thing’ being measured (referred to as the dependent variable). A true experiment aims to measure effect by trying to eliminate alternative explanations of observed relationships between variables. In our software example, the independent variable would be Software X and the dependent variable would be language learning development. It is important to understand that in the social sciences it may be impossible to capture the true dependent variable, which in our case is language development. So a proxy of that variable is needed (such as a language proficiency test), which needs to be appropriately justified as a valid measure of the dependent variable. In intervention studies, an experiment usually embodies a pre-test/post-test design, where the dependent variable is measured before and after the intervention. If time is considered to play an important role, a delayed post-test might also be used to ascertain whether the effects are short term or more sustained.

Essential features of true experiments require: (1) random allocation of participants to experimental conditions and (2) full experimenter control over the independent variables and other external variables. Control groups are often used in experimental research to compare the results of a group receiving the experimental condition (in our case Software X) to a group that does not receive it. To avoid confounding variables or factors, the only way to claim that an observed change is due to the independent variable (e.g. Software X) is to make that variable the only difference between the experimental groups. External variables in the software example could be anything external to the software that could affect language learning gains, such as participants’ aptitude, existing differences in language proficiency, language learning history, motivation, age, teachers and so on. Thus, true experiments often require laboratory-like conditions.

Not only are true experiments difficult to carry out in social science research, they can be seen as problematic in that they may not authentically capture the effects of the independent variable in real-life settings, where external variables are a real part of participants’ everyday lives. To remove these variables could also be seen to reduce the ecological validity of a study. Ecological validity refers to the extent at which the control a researcher exerts over a study still remains representative of the real world. Moreover, in educational research, a researcher may not be able to randomly assign participants to groups and may need to settle for roughly matched classes. In these cases, researchers can choose to adopt a quasi-experimental design.

Quasi-experiments are not required to meet the two conditions of a true experiment (random assignment and full experimenter control) but rather try to get as close to them as possible. In quasi-experimental research the control group is often referred to as a comparison group due to ethical issues arising from a design which requires this group to receive no treatment at all. In our software example, rather than receiving no language instruction at all, the comparison group might continue learning the language via the existing methods of teaching. In some quasi-experiments no comparison group is used, especially when it is neither feasible nor ethical to have one. However, researchers must be aware of the limitations of such a design when attributing observed changes to the intervention.

‘Before’ measures are also a way for researchers to gain control over their experimental condition. For example, a researcher might collect data on students’ aptitude, motivation, previous learning experiences and current language knowledge to see whether these initial differences might explain differences between the two groups. In our software example, good before measures would tell a researcher whether the two groups (those that received Software X and those that did not) were comparable in the first place. Good before measures could lead to statistical analysis to measure how much of the difference could be predicted by other variables (e.g. motivation or aptitude) and how much by the software.

If external variables are thought to create too much ‘noise’ in the data, a researcher could decide to adopt a same-subject design, where the same ‘subjects’ (people) receive all treatments (e.g. Software X and the regular teaching method). The advantage of a same-subject design is that individual differences of participants are the same for each treatment. The disadvantage of same-subject designs is the risk that each treatment may be affected by the subjects’ previous experience. That is, the experience of earlier experimental condition (e.g. using the software) could affect the outcome on later encountered conditions (e.g. studying in the usual manner). If treatments can affect or ‘leak into’ each other in terms of impacting the dependent variable, this is not a good design to use. If same-subject designs are used with multiple interventions or treatments, it is good practice to switch the order of the treatments for some participants (if possible) to counter such effects. Readers who are interested in knowing more about experimental designs in applied linguistics research are encouraged to consult Gass (2015) or Phakiti (2014) for a good overview.

Experimental research can also take the form of other types of design, where variables are manipulated by the researcher to understand their relationships and effects. These types of experiments are explored further in relation to the various tasks and tests used by a researcher to elicit language in order to measure the effect of a manipulation.

Field research

While experiments aim to control for external variables in the study of a construct, field research, sometimes referred to as naturalistic research, aims to embrace these variables as an important part of the phenomena under investigation.

The social and educational world is a messy place, full of contradictions, richness, complexity, connectedness, conjunctions and disjunctions. It is multilayered, and not easily susceptible to the atomization process inherent in much numerical research. It has to be studied in total rather than in fragments if a true understanding is to be reached. (Cohen, Manion & Morrison, 2018: 288)

Naturalistic research, therefore, aims to research the world in its natural setting – thus, embracing the world’s ‘messy’ nature. Field research is an approach to conducting research in the ‘real world’ and is not an exclusive category of research with strict parameters in itself. As a result, field research can include other approaches to research, such as case studies or ethnographies (discussed later in this chapter).

Field research is sometimes considered the opposing design to classical experimental research, and indeed many elements of field research are positioned as ideologically opposite. While experiments try to control for variables, seeing them as contaminating the design, field research embraces them, seeing them as central to understanding the context under investigation. In our example of researching Software X, field research would be less interested in measuring the effects of the software in terms of language gains measured by proficiency tests (although this could be part of the data it collects) but would be more interested in entering classrooms where the software is being used to understand how teachers and learners are engaging with it to gain a rich understanding of how it is being implemented and received. Field research usually requires flexible uses of multiple data collection methods such as (in our software example) interviews with teachers and students, observations of its use, curriculum documents related to its implementation or the collection of test scores in the observed classes.

Because field research requires direct engagement of the researcher with the research setting, the nature of the researcher’s role is an important consideration. This role can generally be placed into five categories on an insider–outsider continuum:

1 Complete participant (e.g. enrolled in the class as an actual student)

2 Participant (e.g. acting in the role of a participant during research)

3 Observer as participant (e.g. working alongside participants during research)

4 Observer (e.g. sitting in the corner of a room, taking notes)

5 Detached observer (e.g. video- or audio-recording the classroom)

The positionality of the researcher during fieldwork has implications for the study design and also how the data should be interpreted later. If a researcher presence is likely to impact the activities being observed, the researcher should try to be as detached as possible. If the events being observed require the researcher to take an active role, then a participant role may be more appropriate. Researcher roles during observation are discussed further in Chapter 7.

Case study research

A case study research design involves the in-depth and contextualized study of the ‘particularity and complexity of a single case’ (Stake, 1995: xi) or multiple cases. In social science research, cases are primarily people, but in applied linguistics a case can also be positioned as a class, a curriculum, an institution, a speech community, a piece of text or a collection of text types.

Case study research, with its in-depth examination of a single case (or small number of cases), generally falls into three case types:

1 An intriguing case, where the peculiarities of the case are the primary focus of research.

2 A typical case, where the event is the primary focus, and the case is of secondary importance.

3 A multiple case, where the event is the primary focus and multiple perspectives are required to capture variability.

In our Software X example, an intriguing case could include a class where the software was reported to work extremely wel...