Connectionist Models of Memory and Language (PLE: Memory)

Joseph P. Levy, Dimitrios Bairaktaris, John Bullinaria, Paul Cairns

• Publisher: Psychology Press
• Year: 2014
• Print ISBN: 9781848723771
• eBook ISBN: 9781317744672
• Topic: Psychology
• Subtopic: Cognitive Psychology & Cognition
• Index: Psychology

Connectionist modelling and neural network applications had become a major sub-field of cognitive science by the mid-1990s. In this ground-breaking book, originally published in 1995, leading connectionists shed light on current approaches to memory and language modelling at the time.

The book is divided into four sections: Memory; Reading; Computation and statistics; Speech and audition. Each section is introduced and set in context by the editors, allowing a wide range of language and memory issues to be addressed in one volume.

This authoritative advanced level book will still be of interest for all engaged in connectionist research and the related areas of cognitive science concerned with language and memory.

SECTION I

MEMORY

Joseph P. Levy

Since the very beginning of interest in them, connectionist networks have been used at least as metaphors for human memory, if not as explanatory models. It is not difficult to pick out some of the historical highlights in the development of models of neural network memories. Hebb (1949) suggested that a short-term trace was held in the activation of a neural circuit while long-term information was held in the synaptic weights. Willshaw et al. (1969) were concerned to describe the formal properties of the information storage in a biologically plausible simple net. Hopfield (1982, 1984) demonstrated how to train a network that stored its memories as stable points in its dynamics – popularizing the theoretical jargon of attractors, basins of attraction, and optimizing training algorithms so as to store maximum information. Rumelhart & McClelland (1986) demonstrated the potential richness in the psychological explanatory power of simple attractor networks that could be trained using some form of learning algorithm. In particular, the way that a partial or noisy cue to an attractor network will, if within the appropriate basin of attraction, lead to the intuitively satisfying “recall” of the appropriate information. There has also been a great deal of work on connectionist models of various aspects of cognitive processing. These often need to appeal to notions of information storage, learning and memory (e.g. Elman & McClelland 1988, Seidenberg & McClelland 1989). The success of these processing models adds to the attraction of using connectionist systems to model memory since it seems likely that memory and processing are intimately connected.

It follows that neural networks and memory are natural partners because of the intuitively satisfying way that networks can learn and store multiple stimuli in the same set of weights; because of the properties of content addressability and noise resistance made possible by nets with attractor dynamics; and because of related successes in connectionist models of cognitive processes.

The chapters in this section describe some of the latest work on the successes and failures of this enterprise. Relationships between memory and processing are highlighted in Chapter 1 on short-term memory for verbal sequences and Chapter 2 on the manner in which groups of stimuli can be temporarily “chunked together”. Chapter 3 covers important ideas on how memory abilities might develop, and the links between connectionist models and the mathematical memory model literature are discussed. A range of problems that back-propagation networks have in modelling human memory data – “catastrophic interference”, “catastrophic discrimination” and unrealistic “hypertransfer” – are discussed in Chapters 4 and 5. This section ends with a discussion of models of the relationship between connectionist short- and long-term stores (Ch. 6).

In Chapter 1, Glasspool describes the Burgess & Hitch (1992) model of the articulatory loop from the working memory literature (e.g. Baddeley 1986). The model is successful at describing the kinds of order errors, word length effects and the detrimental effect of phonemic similarities between words that occur in the working memory literature. Glasspool describes an extension to the model that copes with lexicality effects – the fact that this component of short-term memory works better with words than non-words. The chapter also demonstrates the utility of the simple competitive queue mechanism of Houghton (1990).

In Chapter 2, Bairaktaris describes a low-level modular mechanism for “temporal chunking” serially ordered input sequences. It uses a recurrent back-propagation architecture described by Zipser (1991) to preprocess an input stream into chunks of temporally contiguous items. Bairaktaris stresses the ubiquitous need for this kind of mechanism throughout cognition.

In Chapter 3, Brown, Preece and Hulme describe the links between the current connectionist and mathematical techniques for modelling human memory. They argue that the different approaches have complementary strengths and weaknesses. They stress the need for a combination of the insights from psychological models of the development of memory with those from mathematical models of memory.

In Chapter 4, Sharkey & Sharkey describe the problem of “catastrophic interference” in back-propagation networks, where newly learned information completely overwrites previously learned information. They then describe the complementary problem of “catastrophic discrimination”, the inability to distinguish items that have been learned from new ones. They stress the necessary trade-off between the two problems.

In Chapter 5, Murre demonstrates that back-propagation not only suffers from catastrophic interference but also from “hypertransfer”, i.e. that in some circumstances performance on a set A actually improves when learning a second set B. The learning transfer effects are in disagreement with human learning data. Murre goes on to show that two-layer networks do not suffer from excessive transfer and are in fact in very close accordance with the human interference data as summarized in the classic paper by Osgood (1949).

In Chapter 6, Levy and Bairaktaris survey connectionist models of the interaction between short- and long-term stores. They describe systems with separate components and ones where each connection has both a short- and a long-term weight. They describe their own architecture of dual-weight connections where each weight acts independently. The chapter finishes by speculating on possible future modelling applications for this kind of architecture.

References

Baddeley, A. D. 1986. Working memory. Oxford: Oxford University Press.

Burgess, N. & G. L. Hitch 1992. Towards a network model of the articulatory loop. Journal of Memory and Language 31, 429–60.

Elman, J. L. & J. L. McClelland 1988. Cognitive penetration of the mechanisms of perception: compensation for coarticulation of lexically restored phonemes. Journal of Memory and Language 27, 143–65.

Hebb, D. O. 1949. The organisation of behaviour. New York: John Wiley.

Hopfield, J. J. 1982. Neural networks and physical systems with emergent computational abilities. National Academy of Sciences of the USA, Proceedings 79, 2554–8.

Hopfield, J. J. 1984. Neurons with graded response have collective computational properties like those of two state neurons. National Academy of Sciences of the USA, Proceedings 79, 2554–8.

Houghton, G. 1990. The problem of serial order: a neural network model of sequence learning and recall. In Current research in natural language generation, R. Dale, C. Mellish, C. M. Zock (eds), 287–319. London: Academic Press.

Osgood, C. E. 1949. The similarity paradox in human learning: a resolution. Psychological Review 56, 132–43.

Rumelhart, D. E. & J. L. McClelland 1986. Parallel distributed processing. Cambridge, Mass: MIT Press.

Seidenberg, M. S. & J. L. McClelland 1989. A distributed, developmental model of word recognition and naming. Psychological Review 96, 523–68.

Willshaw, D. J., O. P. Buneman, H. C. Longuet-Higgins 1969. Non-holographic associative memory. Nature 222, 960–2.

Zipser, D. 1991. Recurrent network model of the neural mechanism of the short-term active memory. Neural Computation 3, 179–93.

CHAPTER 1

Competitive queuing and the articulatory loop

David W. Glasspool

Introduction

This chapter reviews the psychological evidence for the articulatory loop. Two neural network implementations of the articulatory loop model are then discussed. The first is the model of Burgess & Hitch (1992). The second is an extended model which improves on that of Burgess & Hitch in its ability to operate with non-words as well as words, in its better simulation of the typical forms of order error made by human subjects, and in its ability to show both primacy and recency effects.

Both models use a competitive queuing architecture in order to achieve serial recall with single-shot learning. It is argued that competitive queuing is a particularly suitable paradigm for modelling the serial recall of verbal material due to its straightforward explanations for commonly observed error types and serial position effects.

The articulatory loop

There is evidence from studies on normal subjects and on neuropsychological patients that a specific dissociable system underlies the verbal component of short-term memory (e.g. see Baddeley 1990). This is the system presumed to be used in the immediate recall of word lists (span tasks), and which Baddeley & Hitch (1974) identified as the articulatory loop in their “working memory” model of short-term memory. The working memory framework proposes a modular system mediating short-term memory, in which a limited-capacity central executive is able to off-load information for short-term storage to at least two subsystems, the visuo-spatial scratchpad for spatial material, and the articulatory loop for verbally coded material.

The articulatory loop has been characterized by Baddeley (1986) as consisting of two components, a phonological store holding a rapidly decaying memory trace in the form of articulatory (motor) information, and a rehearsal mechanism, presumed to be formed from parts of the speech input and output mechanisms, which enables the system to maintain information for longer periods by continually refreshing the store’s contents.

Auditorily presented material is presumed to gain immediate entry to the phonological store. Visually presented material gains access when it is articulated. In order to refresh the rapidly decaying trace it is necessary to re-articulate the store’s contents. The rehearsal process places a limit on the amount of information which the store can hold – this is simply the amount of information which can be articulated by the speech mechanism in the time it takes for the trace in the store to decay to the extent that it is unusable. Notice that this limit is imposed by the fact that the process of articulation is inherently sequential, and not by a limited number of “chunks” or “slots” in the store. The store itself may be thought of as a constantly decaying trail left behind by all articulation, such that a speaker always has access to the last few seconds of speech.

Baddeley (1986) finds a limit of 1.5–2 seconds on the persistence of this trace, so the articulatory loop has a capacity set by the amount of speech which can be articulated in this time. This relationship between speech rate and span has been demonstrated several times (e.g. Baddeley et al. 1975, Ellis & Henneley 1980, Hulme et al. 1991).

The articulatory loop is specified at a very general level, but has nonetheless proved to be a very robust basis for the explanation of much of the data on human performance in this area. The word length effect (span is reduced for lists of longer words) and phonemic similarity effect (span is reduced for lists containing phonemically similar words) under different conditions of presentation modality and suppression are well accounted for (Baddeley 1986), the word length effect being due to articulatory rehearsal of the contents of a store which has a rapidly decaying trace, while the phonemic similarity effect is due to interference between similar items within the store. The model also predicts such phenomena as the effects of cross-linguistic differences in articulation rate on span. Ellis & Henneley (1980), for example, found that Welsh-speaking subjects had a shorter digit span than English speakers. The difference disappeared when the longer time required to articulate the Welsh digits was taken into account, as would be expected from a system with a limit on the articulatory duration of the material it can hold.

However, the model is specified at too general a level to account for other well established characteristics of verbal short-term memory. This is largely due to the fact that no architecture is specified for the “store” component. The representation of serially ordered information in a neural system is a far from trivial task, and it might well be expected that the need to store order information would place strong constraints on the architecture of the store, and hence on the detailed behaviour of the system. In particular, the basic articulatory loop model as outlined above fails to account for the following:

(a) The types and proportions of errors in immediate serial recall tasks. Between 70% and 80% of errors made by subjects in tasks involving the immediate recall of a list involve the order of items in the list rather than the identity of the items (Aaronson 1968). As Lashley (1951) pointed out, the form of typical errors argues against any form of “chaining” in the representation of stored lists. In fact, although the order of items may be incorrect they still tend to be produced only once, so a typical error is the exchange of two items in the list. This type of error is very difficult to explain on any intuitive symbolic account of the structure of the store...

Table of contents

• Cover Page
• Half Title page
• Title Page
• Copyright Page
• Original Title Page
• Original Copyright Page
• Contents
• Preface
• Acknowledgements
• Contributors
• Section I Memory
• Chapter 1 Competitive queuing and the articulatory loop
• Chapter 2 Temporal chunking and synchronization using a modular recurrent network architecture
• Chapter 3 Learning to learn in a connectionist network: the development of associative learning
• Chapter 4 Interference and discrimination in neural net memory
• Chapter 5 Transfer of learning in back-propagation and in related neural network models
• Chapter 6 Interactions between short- and long-term weights: applications for cognitive modelling
• Section II Reading
• Chapter 7 Self-learning and connectionist approaches to text-phoneme conversion
• Chapter 8 Reading exception words and pseudowords: are two routes really necessary?
• Chapter 9 Neural network models of reading: solving the alignment problem without Wickelfeatures
• Section III Computation and Statistics
• Chapter 10 Cortical neurocomputation, language and cognition
• Chapter 11 Neural networks: the new statistical models of mind
• Chapter 12 Acquiring syntactic information from distributional statistics
• Section IV Speech and Audition
• Chapter 13 Onset/offset filters for the segmentation of sound
• Chapter 14 Time-warping tasks and recurrent neural networks
• Chapter 15 Bottom-up connectionist modelling of speech
• Chapter 16 Interactive models of lexicalization:some constraints from speech error, picture naming, and neuropsychological data
• Index