Quantitative Research Methods in Translation and Interpreting Studies

Christopher Mellinger, Thomas Hanson

• Publisher: Routledge
• Year: 2016
• ISBN:
  9781317299226
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Quantitative Research Methods in Translation and Interpreting Studies encompasses all stages of the research process that include quantitative research methods, from conceptualization to reporting. In five parts, the authors cover:

• sampling techniques, measurement, and survey design;

• how to describe data;

• how to analyze differences;

• how to analyze relationships;

• how to interpret results.

Each part includes references to additional resources and extensive examples from published empirical work. A quick reference table for specific tests is also included in the appendix.

This user-friendly guide is the essential primer on quantitative methods for all students and researchers in translation and interpreting studies. Accompanying materials are available online, including step-by-step walkthroughs of how analysis was conducted, and extra sample data sets for instruction and self study: https://www.routledge.com/9781138124967. Further resources for Translation and Interpreting Studies are available on the Routledge Translation Studies Portal: http://cw.routledge.com/textbooks/translationstudies.

Part I

Preparing

As described in the introduction to the volume, this book is divided into several parts that correspond to stages of the research project. Part I focuses on the initial preparation stage. The best research projects have as their foundation a well-motivated research question or hypothesis. Without understanding the nature of the research question(s) and purpose of the project, adequate samples or measurements cannot be obtained, data cannot be collected, and statistical tests cannot be appropriately selected or interpreted.

Therefore, Chapter 1 focuses on generating appropriate research questions and hypotheses. We also distinguish several common research designs. Chapter 2 is dedicated to sampling and ethics. For research involving human participants, ethical considerations related to the recruitment, selection, and inclusion of participants are intrinsically linked to sampling practices. Chapter 3 then addresses measurement and the notions of reliability and validity. Part I closes with a brief treatment of survey design and technology issues related to measurement.

1 Research questions, hypotheses, and design

This chapter introduces the basics of quantitative research methods. In order to conduct meaningful research, research questions and hypotheses require considerable thought and should be aligned with the extant literature. Personal experience or general observation may initially spark interest in a particular topic, and a review of the related scholarly work might reveal a potential area of investigation. Both research questions and hypotheses serve as the foundation of scholarly inquiry. Furthermore, research questions determine what data are needed to answer these questions and, by extension, how the data should be collected. These data will eventually determine the analytical method to be used and the types of conclusions that can be drawn.

Research questions

Investigators often begin to explore an area of research with a certain topic in mind. For instance, they may be interested in the impact of technology on the translation process or how a speaker might influence the performance of an interpreter. These questions are broad and do not make any predictions about the potential relationship between these two ideas; that is to say, they are non-directional. A broad question may also be broken down into related sub-questions that focus on more specific aspects of the overarching question.

To help structure the research project, researchers should identify a theoretical framework within which the research questions will be explored. At the highest level of abstraction, a research question involves a collection of concepts that are related to the area of interest. For the previous interpreting example, the investigator would need to explore various aspects of the interpreting task that are related to the research question. As meaningful relationships between these concepts are established, a construct begins to emerge. A rough definition of a construct is a group of concepts linked in a theoretical model. Constructs help researchers to refine their questions and provide structure to their investigations.

As the research process continues, investigators must further refine constructs into measurable and observable variables. To do so, constructs must be operationalized so that a research design or protocol can be developed to investigate the variables of interest. Operationalization entails a strict definition of the constructs so that they can be measured and tested. Researchers often draw on previous definitions of concepts or constructs to guide operationalization, and this linkage to previous research can help contextualize study findings in relation to other studies.

Two basic orientations to research can be adopted: inductive and deductive.¹ Generally speaking, deductive reasoning adopts a top-down perspective, in which a general theory leads to testable predictions (Evans 2013). In the interpreting example, a model on interpreting effort or performance might generate research questions that investigate specific aspects of the theory. Conversely, inductive reasoning attempts to use observations from a study as the foundation for theoretical development. Quantitative research tends to adopt a deductive line of reasoning since empirical and experimental observation can be used to test theories related to a specific object of analysis. This theory-driven approach to research allows investigators to evaluate the suitability of models or theories to describe specific aspects of translation and interpreting (T&I).

In a study, researchers must operationalize two types of variables: independent and dependent variables. In an experimental setting, independent variables are those that are manipulated or controlled by the researcher. The independent variable can also be referred to as the treatment variable. Returning to the interpreting example, if the investigator were to change the speaker’s delivery rate and measure performance under different conditions, then the delivery rate would be considered an independent variable. It is important to distinguish between the independent variable, which is delivery rate, and the levels of the independent variable, which are the various rates. In this simple study, there is only one independent variable, but it includes several possible levels (e.g., slow, average, and fast speeds). More than one independent variable is possible in a study, but great care must be taken when introducing multiple independent variables. In non-experimental situations, the independent variable is not controllable by the researcher. In those cases, the independent variable often is referred to as a causal, treatment, input, or explanatory variable.

The dependent variable is the phenomenon that the researcher observes and measures but does not control. In many cases, the goal of research is to demonstrate a causal effect of the independent variable on the dependent variable. Causation, however, cannot be fully established in a single study. Outside of a fully-controlled experiment, researchers can establish that a correlation or relationship exists between variables, but a cause-effect relationship cannot be determined. In the interpreting example, some measure of the interpreter’s performance (e.g., disfluencies) would be the dependent measure that is presumably affected by the delivery rate (i.e., independent variable).

The primary variables of interest in a research question are the independent and dependent variables; however, additional variables can affect their relationship. Terminology varies widely for these additional variable types, which are sometimes subsumed under the term extraneous variables. For instance, a confounding variable is one that potentially alters or influences the relationship between the independent and dependent variables. As an example, in a study of text readability and translation performance, environmental factors like noise or lighting could affect the participants’ work. If performance differs among participants, but the noise level and lighting varied when they completed the task, then these variables will confound the results. Consequently, the researcher cannot determine what caused the change in performance—the independent variable of text readability or the variation in the experimental setting. To eliminate confounds, researchers can either control the experimental setting or introduce one or both of these as a control variable in the statistical analysis. Model selection and the inclusion of control variables is a topic that belongs primarily to linear regression (see Chapter 12).

Research hypotheses

The difference between research questions and hypotheses is largely a matter of degree. Research questions are phrased in a broader sense, postulating the possibility that a relationship exists between variables. In contrast, research hypotheses presuppose that a relationship exists between the independent and dependent variables and, in some cases, predicts the nature or direction of the relationship.

Research hypotheses can be either non-directional or directional. A non-directional hypothesis would state, for example, that faster rate of delivery has some effect on the number of interpreter false starts. In this case, the researcher suggests that a difference exists between these two conditions, but does not provide a prediction. In contrast, a directional hypothesis would predict the actual relationship, stating that a faster delivery rate increases the number of interpreter false starts.

The distinction between directional and non-directional hypotheses is important because it influences the way in which data are analyzed. Without compelling evidence from previous studies or without a specific theory to support directional hypotheses, non-directional analysis should be preferred. In this volume, the emphasis is placed on non-directional hypotheses. This decision is based on the predominance of this type of statistical analysis and for simplicity of presentation.

Researchers refer to the statistical tests of non-directional hypotheses as two-tailed tests, while directional hypotheses are called one-tailed tests. The underlying mathematical processes of these tests do not change based on the nature of the hypotheses. The only change occurs in the final interpretation of the results. Therefore, a researcher who desires to test a directional hypothesis can follow the guidelines of this volume with only a minor shift in interpretation. We stress, however, that such one-tailed tests are not as common as non-directional hypotheses, especially for exploratory research. Finally, thanks to statistical computer software, tests are easily adapted for a one-tailed test.

Null and alternative hypotheses

To this point, the focus has been on the development of a research question or hypothesis; however, these are not tested directly. Instead, quantitative research relies on the notion of refutability or falsifiability and uses this concept to test a second, related hypothesis known as the null hypothesis. Simply put, the null hypothesis states that no difference or relationship exists between the variables that are included in the research question or hypothesis. In the interpreting example, the null hypothesis would state that there is no relationship between the number of interpreter false starts and the speed of the speaker. Rather than setting out to prove the research hypothesis, statistical testing instead seeks to reject the null hypothesis.

This approach provides the starting point for statistical hypothesis testing. However, before proceeding to testing the null hypothesis, an important conceptual distinction needs to be made. Note that nowhere in this volume will we refer to proving a hypothesis to be true or false. Instead, we will discuss this in terms of “rejecting the null hypothesis” or “failing to reject the null hypothesis.” One study can never prove the existence of a difference or a relationship between variables. Particularly in the social sciences, theories are developed through ongoing inquiry involving multiple studies on a topic.

Similarly, a null hypothesis is never proven nor accepted as true. Failure to reject a null hypothesis represents a lack of evidence against it, which is not the same thing as evidence for it. An analogy is often drawn to the legal system, in which lack of conviction in a court of law does not constitute proof of innocence, merely lack of enough evidence for conviction. The same idea is expressed in the pithy saying that absence of evidence is not evidence of absence. Being built upon probability theory, mathematical statistics is not designed to generate conclusive proof and always contains a probability of error. Therefore, particular language must be employed in reporting statistical test results to portray the evidence correctly.

Several conventions exist for the notation of statistical hypotheses. Throughout this volume, we will use H₀ (pronounced H-nought) for the null hypothesis and H₁ for the alternative hypothesis. The statistical notation related to inferential statistics that use the null and alternative hypotheses will be presented in each chapter.

Research design

The manner in which research will be conducted follows from the research questions or hypotheses; the type of research must be in support of the project. Several types of research are conducted in T&I studies, including experimental, quasi-experimental, and observational research. Researchers must weigh the advantages and drawbacks of each type of research to determine which design is best for their specific research questions or hypotheses. Alternately, researchers can employ several methods, known as triangulation, to provide a varied and complementary perspective. The use of several methods, measures, or theoretical frameworks to examine a particular phenomenon or behavior improves the robustness of study and can corroborate findings across the various measures (Keyton 2014).

Experimental research

Research that involves the specific manipulation of conditions or variables in a controlled environment would be considered experimental research. Two major aspects of a project characterize experimental research: control and random assignment. In an experiment, the researcher controls and manipulates the independent variables under specified conditions. Control of the independent variable(s) and the experimental conditions is one way for researchers to avoid potential confounding variables. A criticism of experimental studies is precisely the contrived nature of many experimental tasks and the artificial environment in which they are conducted. This potential disadvantage must be weighed against the benefits derived from controlling the variables.

Random assignment is the second characteristic of experimental research designs. When participants are enrolled in a study, researchers randomly assign them to various conditions. In some cases, the conditions would be a control or treatment group; in other cases, the participants receive various treatments or perform under different conditions. Random assignment aims at uniformity between the various groups and helps prevent confounding variables from affecting the results. Random assignment is distinct from random sampling, discussed in Chapter 3, but the two concepts work in cooperation to assert that observed effects in an experiment are the result of the experimental treatment or condition.

Two common experimental research designs are independent and repeated measures designs. The distinction between these two designs is whether participants take part in only one condition of the study or if they are measured under multiple conditions. A participant enrolled in an independent measures experiment would be placed into either the control or treatment group, and would only complete the task specific to that group. One benefit of this design is that participants are not unnecessarily burdened with multiple treatments, and attrition is potentially reduced. A drawback, however, is the need for a greater number of participants to enroll in each of the study groups.

Repeated measures designs, in contrast, enroll participants into multiple conditions or treatment groups. For instance, an experiment that is investigating interpreting in various settings may have the same interpreter perform a task in multiple conditions. Another variant of repeated measures design is to measure a participant at multiple points in time. A benefit of any repeated measures design is that the amount of variability is mitigated since each participant serves as his or her own control. Many factors are held constant when the same participant is measured more than once. Because of the lower variance, fewer participants are required for the study and statistical properties are improved in many cases. A drawback, however, is that participants need to complete multiple tasks; in this scenario, there is a greater likelihood of missing data as the result of attrition, or the possibility of additional confounding variables, such as fatigue, order effects, or carryover effects.

Quasi-experimental research

Researchers are not always able to control the independent variables; at times these variables vary naturally so that ...