Rating EFL Written Performance

Katalin Bukta

• Publisher: De Gruyter Open Poland
• Year: 2014
• Print ISBN:
  9788376560786
• eBook ISBN:
  9783110399288
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

The study focuses on the investigation of the process during which raters of EFL written performance make their decisions. It consists of a pilot and a main study, each of which concentrates on assessment of writing. The rationale is to detect the decision-making processes that raters follow, which can be used for training raters, and with which the reliability of rating can be improved. The pilot study is based on data collected during a large-scale language proficiency assessment of two age groups from learners of English and German languages. Raters were asked to think aloud during the rating task. Data was then transcribed and analysed. The participants in the main study were novice raters, who produced verbal protocols. 37 EFL teacher trainees took part in rater training and practised think-aloud protocol production. Then, they evaluated ten compositions written by EFL learners and verbalised their thought processes. The verbal protocols served as a basis for data collection. The analysis of data resulted in the conclusion according to which more reliable and objective assessment is possible when evaluating written performance.

PART I

An Overview of the Literature on L2
Writing and Assessment

The first part of the book intends to provide an overview of the relevant literature on writing and assessment from Chapter One to Chapter Five.

Chapter 1

Writing Ability and L2 Writing
Instruction

Introduction

The way people interact has always been central to research, since the need to communicate effectively is crucial. Modern age is characterised by advanced technology, especially in communication, however, we cannot exist without written communication. Before discussing how writing ability can be assessed, we have to have a look at the features of writing ability, what characterises the ability to write in one’s L1 and what skills are needed for being able to express ourselves in writing in a new language (L2). This chapter first deals with the nature of the writing skill in L1 and L2 focusing on elements constituting language ability. The terms foreign language and second language are used interchangeably from now on and are labelled as L2. Then, frameworks of communicative language ability are presented briefly to highlight the linguistic aspect of language competence. In order to be able to measure language ability we have to have a clear picture of its components and the relationships between them. Writing ability in L1 and L2 is compared through looking at the way skilled and unskilled writers compose texts. In addition, the influence of reading on writing processes and the role of L1 in L2 writing are presented. Writing ability is strongly related to education and writing skill should be taught; therefore, a discussion of writing instruction both in L1 and L2 is relevant. The role of writing in L2 instruction is important, as language proficiency develops using writing as a learning tool, especially in the case of adult language learners (Weissberg, 2000). Writing skills are transferable from L1 to L2 writing and the transfer takes place during learning (Wolff, 2000). The discussion focuses on the main characteristics of L2 writing instruction, and, finally, main issues of L2 writing instruction are introduced briefly.

1.1 The Writing Skill in L1 and L2

The ability to write is not innate. Compared to listening and speaking, people need to reach a certain level of cognitive development before they can acquire writing skills. The skill comprises of several elements; it is not sufficient to learn how to form letters on a piece of paper with the help of a pen or a pencil. Virtually anybody who does not have a physical deficiency can learn to speak, but writing skill development is much more complex, as it is part of general literacy skills, involving reading comprehension as well. Although speaking and writing are closely related, writing means more than merely recording spoken language (Weissberg, 2000).

Proficiency in a language can be described by making a distinction between four language skills and dividing them up according to the channel of communication: two of them, speaking and listening are oral skills, and two, reading and writing are written skills. It is also possible to differentiate between productive and receptive skills according to the mode: the former two involve language production (speaking and writing) and the latter ones involve receptive skills (reading and listening). Both L1 and L2 writers have to have all these skills at their disposal in order to be able to make the right decisions when choosing from the language store. It follows that writing is a productive language skill, which has both similar and different features in one’s L1 and L2 (Harmer, 1991, p. 15). However, this distinction according to the channel and mode seems to be vague for understanding the processes involved in language use (Bachman & Palmer, 1996, p. 75). Although mental processes of language production are not easy to follow, several models have been proposed to explain the process a writer goes through in L1 and L2 writing development. These models are introduced in the followings to look into the nature of writing ability in L1 and L2 and their relationship.

1.1.1 Theoretical Models of Written Text Production

In order to have a better insight into L2 written language production, it is worth considering the models that attempt to describe how L1 writers approach and then perform a writing task. The models examine different elements of the writing process, such as the task, its environment, the writer and the audience; as well as provide an explanation of the relationship between these elements and the way they interact with each other. Thus, they facilitate the understanding of the cognitive processes, the knowledge needed and other underlying factors are easier to understand (Weigle, 2002, p. 23). The theoretical models presented below shed light on the main cognitive processes that writers follow when producing texts.

The model proposed by Hayes and Flowers (1980) includes three main cognitive processes that play a role in written language production: planning, translating and reviewing. Planning consists of organising, goal setting and generating; these steps are closely related to the writer’s knowledge of the topic and strategies which enable him to organise thoughts coherently. During the translating phase the writer retrieves relevant information from long-term memory and transforms it into language. Translating in the model means retrieving thoughts from memory and turning them into language. Finally, the text is reviewed to improve it in the translation phase with the help of reading and editing skills. The order of these processes is not linear, the writer can go back to some stages to improve the text, but most probably the steps follow each other starting from goal setting through planning, production to revision. These three main cognitive processes in the Hayes and Flower (1980) writing model are closely related to the writer’s long term memory, where the knowledge of the topic and the audience is stored, and there are writing plans from which the writer can choose. When the writer is assigned a task, the task environment contains the topic, the audience and motivating cues with the text produced so far. The model centres on the role of planning and revising, and less attention is paid to sub-processes. Nevertheless, the recognition of the recursiveness of the writing processes is important and has served as a basis for further research.

The model developed by Hayes (1996, cited in Weigle, 2002, pp. 24-25) discusses the processes from two aspects: the task environment and the writer. In the model the focus is rather on the individual, whose motivation, working and long-term memory, and cognitive processes are examined in detail. Task environment comprises physical and social factors, which are the people involved including audience and the text, and the medium of writing. The role motivation and affect play in the model is considerable and they are related to the cognitive processes of text interpretation, reflection and text production. It means that the individual’s success in performance depends on motivational factors. The information about the task and topic is stored in long-term memory, whereas working memory stores three types of information: verbal, coded and conceptual. These interplay with cognitive and motivational processes at different stages of text production. The other important feature that is highly relevant to assessment and instruction is that there are several reading types involved in the writing process: reading to evaluate, reading source texts and reading instructions. It follows that if the writer does not comprehend written texts properly, he cannot perform writing tasks. Hayes discusses the relevance of several types of reading in written text production: reading instructions, sources and the text during revision are the most important that need consideration.

Bereiter and Scardamalia introduce a two-model approach to writing (Bereiter & Scardamalia, 1987). They investigate the writing process by examining the differences between skilled and unskilled writers. Moreover, they introduce the notions of “knowledge telling” and “knowledge transformation”: the first means simply recording speech, whereas the second relates to composing new language. The two terms refer to the following differences in text production: knowledge telling is when language is recorded with no or little planning, translation or other text production processes, while knowledge transformation involves all cognitive processes discussed above to come up with a new text. The model builds on the assumption that while people can learn to speak it is not evident that they can also learn to write. The main difference between unskilled and skilled writers is in the use of strategies: unskilled writers employ fewer and simpler strategies than expert writers. Unskilled writers spend less time on planning and they revise less than expert writers. These features bring writing and speaking closer to each other, but the difference is that while in speaking there is interaction, during which clarifications and amendments can be made, in writing it is not possible. Moreover, the strategies skilled and unskilled writers use show different features, as expert writers’ strategies do not develop from the simpler strategies used by unskilled writers, but they employ different ones. The Bereiter and Scardamalia (1987) two-model approach has informative implications to writing instruction, as it makes a distinction between skilled and unskilled writers; however, the model does not provide sufficient guidance regarding how to turn unskilled writers into skilled ones.

1.1.2 Theoretical Frameworks of Communicative Competence

Theoretical models of written text production explain cognitive processes, but fail to give an account of the linguistic knowledge that is needed for language production. In order to be able to understand the details of language ability, we have to turn to the notion of communicative competence which was introduced by Hymes (1972) in the 1960s and discussed further by Canale (1983) among others. The Bachman and Palmer (1996) model approaches language ability from language testing perspective, whereas Celce-Murcia, Dörnyei and Thurrell (1995) focus on language teaching aspects. The different models attempt to shed light on the elements constituting the notion of communicative competence, and explain the relationship between them. They serve as a basis for better understanding both L2 instruction and testing (Katona, 1995).

Several attempts have been made to describe what it means to be able to use a language and what elements such knowledge consists of. Chomsky (1965, cited in Celcia-Murcia, Dörnyei, & Thurrell, 1995, p. 6) refers first to language constituting of “competence” meaning the knowledge of the rules of the language and “performance” which is the actual use of the language. This rather static perception of language knowledge is further refined by Hymes (1972), who redefined these two elements and added a new, sociolinguistic aspect to it. His model constitutes four elements: formal features, realisation, appropriacy and accepted use (Hymes, 1972, cited in Katona, 1995, p. 69). This model takes context into account and considers it an essential element in language knowledge.

The need to explain how language can be used for communication and to integrate it with other components of language ability resulted in the model developed by Canale and Swain (1980), which consists of three components: grammatical, sociolinguistic, and strategic competences. Canale (1983) proposes a revised version of the framework and he relates it to both foreign language teaching and testing. First, he explains the difference between communicative competence and actual communication saying that knowledge of the language system and skills is needed for actual communication: in his model communicative competence means both the knowledge of the language and the skills or the ability to use it. Canale distinguishes four competence areas which interact with each other and are required in communication. Grammatical competence is the knowledge of the language code, sociolinguistic competence is appropriacy of meaning and form of language, discourse competence relates to cohesion and coherence, and strategic competence which a language user refers to when there is a breakdown in communication or when he wants to make the communication process more effective.

The framework is further refined and explained by Bachman (1990) who tries to show the processes that interact between the elements and the possible application of the framework to language testing. He uses a slightly different term and describes communicative language ability as an interaction of knowledge structures and language competence with strategic competence, psychophysiological mechanisms and the context of situation. Strategic competence is not directly related to language and is viewed as a capacity to operationalise components of language ability. Language competence is then explained as comprising two main competences, organisational and pragmatic, which are further divided into grammatical and textual competences, and illocutionary and sociolinguistic competences respectively. The framework developed in Bachman and Palmer’s model (1996, p. 62) emphasises the relationship between language use and its testing to explain how language ability can be assessed when performing test tasks. The elements of language ability, topical and language knowledge together with personal characteristics interact with strategic competence and are influenced by affective schemata; all these interact with language use and test task. In addition, Bachman and Palmer reconsider strategic competence in Bachman’s model and introduce metacognitive strategies to demonstrate that language knowledge and metacognitive strategies make the language user able to comprehend and produce language.

The model developed by Celce-Murcia, Dörnyei and Thurrell (1995) views communicative competence from a L2 teaching perspective. It derives from synthesis and in some respect is a redefinition of elements of previous models proposed by Canale, Canale and Swain, and Bachman and Palmer. According to the model, communicative competence consists of five main elements: discourse, linguistic, actional, sociocultural, and strategic competencies. Discourse competence comprises elements that are necessary to produce texts: cohesion, deixis, coherence, genre, and conversational structure. Linguistic competence corresponds to descriptive linguistic aspect: syntax, morphology, lexicon, phonology, and orthography (the last two depending on the mode). Actional competence is derived from sociocultural competence in other models and is treated separately in this model and defined for oral communication. It is related to understanding intentions and responding to them appropriately. In addition, it contains the following two broad components: knowledge of language functions and speech acts. The former is further subdivided into interpersonal exchange, information, opinions, feelings, suasion, problems, and future scenarios. Sociocultural competence refers to the knowledge of expressing oneself appropriately applying the following factors: social contextual, stylis...

Table of contents

• Cover Page
• Title Page
• Copyright
• Contents
• Introduction
• Overview of the Book
• Part I An Overview of the Literature on L2 Writing and Assessment
• Part II Investigating Raters’ Decision-Making Processes and Awarded Scores in Rating Hungarian Efl Learners’ Compositions
• Backcover