Parallel Models of Associative Memory

Updated Edition

Geoffrey E. Hinton, James A. Anderson, Geoffrey E. Hinton, James A. Anderson

• Publisher: Psychology Press
• Year: 2014
• Print ISBN: 9780805802696
• eBook ISBN: 9781317785200
• Topic: Psychology
• Subtopic: Cognitive Psychology & Cognition
• Index: Psychology

This update of the 1981 classic on neural networks includes new commentaries by the authors that show how the original ideas are related to subsequent developments. As researchers continue to uncover ways of applying the complex information processing abilities of neural networks, they give these models an exciting future which may well involve revolutionary developments in understanding the brain and the mind -- developments that may allow researchers to build adaptive intelligent machines. The original chapters show where the ideas came from and the new commentaries show where they are going.

1	Models of Information Processing in the Brain

James A. Anderson
Brown University
Geoffrey E. Hinton
M.R.C. Applied Psychology Unit
Cambridge, England

1.1. INTRODUCTION

This chapter introduces some models of how information may be represented and processed in a highly parallel computer like the brain. Despite the staggering amount of information available about the physiology and anatomy of the brain, very little is really known about the nature of the higher-level processing performed by the nervous system. There is no established theory about the kinds of neural activity that occur when we hear a sentence, perceive an object, or form a plan, though data on many fascinating and significant bits and pieces is now available.

An obvious feature of the brain is its parallelism (see Section 1.5 for a review of the neurophysiological evidence). This parallelism is a major reason for investigating computational models other than the conventional serial digital computer in our attempts to understand how the brain processes information. The concept of parallelism may need some explanation. A system which is parallel at one level of description may well be serial at a higher level. At the level of individual motor neurons, for example, the human motor system is highly parallel. The simultaneous actions of many muscles are necessary for coordinated movement. If, however, the pattern of activity of the whole set of motor neurons is used as a unit of description, the system is strictly serial because only one pattern can exist at a time. Similarly, in a conventional digital computer many electrical events occur in parallel when each machine instruction is executed, but the instructions, considered as units, are executed sequentially. The transition between the parallel and serial levels of description thus occurs at the level of the individual machine instructions.

If many computational operations are performed at once, a system can obviously operate faster. However, one part does not know what the other parts are currently doing because the parts operate simultaneously. This causes serious problems of coordination and lateral information transfer from one part of a parallel system to another. These problems have made it hard to program general purpose computers that execute many instructions at once though numerous special purpose systems have been developed for specific tasks.

We feel that problems of coordination and lateral information transfer are not merely irritating; they are fundamental. They determine the kinds of operations that are easy to implement at the level at which a machine is parallel. These may be much richer than the rather restricted set of primitive operations of a conventional digital computer. For example, the state of activity of a large set of feature-detecting units can determine the state of activity of another large set of units in a single step in a parallel machine.

The idea that a parallel machine may have a different and richer and much more powerful set of primitive operations constrasts sharply with the idea that parallelism should be added on top of existing programming techniques by providing message-passing facilities that allow communication between multiple processors, each of which is a fully fledged conventional computer. The latter approach is obviously a sensible way of extending existing computational techniques, and it is currently under investigation within computer science, but it takes for granted the primitive operations of a conventional digital computer, which are probably an inappropriate computational metaphor for the brain.

As an example of a task for which a conventional computer seems inappropriate, consider the problem of recalling an item when given a partial description of its properties or a description of its relationships to several other items (Norman & Bobrow, 1979). This appears to be a fairly basic human ability. If the partial description is sufficient to identify an item uniquely, the item often just “comes to mind,” with no awareness of any deliberate searching. It is relatively easy to implement this kind of access to items in memory if all the partial descriptions that might be used for access are known in advance. However, human memory does not seem to require this. We can access items from partial descriptions that have not been anticipated. This kind of memory, in which the partial contents of an item can be used to retrieve the remaining contents, is called content-addressable memory. It is a desirable thing to have, but it is very hard to implement in a conventional digital computer (a von Neumann machine). The reason for the difficulty is that the von Neumann machine accesses items in memory by using their addresses (locations in memory), and it is hard to discover the address of an item from an arbitrary subset of its contents. As we shall see, if we abandon the idea that the basic method of retrieving items is via their addresses, we can use parallel computation in systems of interconnected simple elements to achieve content-addressable memory.

Von Neumann machines are based on the idea of a sequential central processor operating on the contents of a passive memory in which data-structures simply wait around to be inspected or manipulated. This conception of memory is shared by most psychologists and is embodied in the spatial metaphors we use for talking about the process of remembering. We think of memory as if it were a filing cabinet or warehouse, and the act of recalling an item is referred to as finding it in memory as if each item were in a specific place and could be found only by going to that place. How else could it be?

The memory models presented in this volume assume a very different basic architecture. Instead of a sequential central processor and a passive memory there is a large set of interconnected, relatively simple processors, which interact with one another in parallel via their own specific hardware connections. Changes in the contents of memory are made by forming new connections or changing the strengths of existing ones. This overcomes a major bottleneck in von Neumann machines, which is that data-structures or programs in memory can only have effects via the sequential central processor, so that it is impossible to mobilize a large quantity of knowledge simultaneously.

A consequence of replacing passive memory by simultaneously interacting units is that the addressing mechanism is replaced by specific hardware connections. The addressing mechanism allows the central processor of a von Neumann machine to access any piece of data, provided the address is known. It thereby allows complex data-structures to be stored in memory by simply making one piece of a data-structure contain the address of the next piece. If one piece contains several addresses, branching structures like trees can easily be stored. Such structures appear to be essential for the implementation of complex representations and computational procedures.

Feldman (Chapter 2, this volume) and Fahlman (Chapter 5, this volume) propose that addresses be replaced by specific hardware connections. Some of the other models in this volume also replace addresses by hardware connections but in a less direct manner. They do not replace a single address by a single hardware connection because they do not use the individual processing units to correspond to items in memory. Instead, items correspond to patterns of activity distributed over many simple hardware units, and the ability of an address to link one item to another is implemented by modifying the strengths of many different hardware connections in such a way that the pattern of activity corresponding to one item can cause the pattern corresponding to the other item (see Section 1.2.3 for details).

The idea that a pattern of activity could represent an item requires some explanation. We use the term distributed representation to refer to this way of coding information. Although the concepts of distributed representation and parallelism are quite different, distributed representation appears to be a particularly appropriate method of coding for a highly parallel machine.

Suppose we wish to build a system that can recognize any one of a number of items. One approach to this problem would be to have one internal unit that would respond when and only when its particular item occurred. An alternative approach would be to have each internal unit respond to many of the possible input items. Provided only one item is presented at a time, it will be represented by the pattern of activity of the internal units even though no individual unit uniquely specifies the input item. Thus a pattern of activity becomes the basic representation of the item. There is no necessary loss of precision or ability to discriminate; it is just that internal operations are now performed in a different way. Instead of a single unit causing particular effects on other internal representations or on motor output the pattern of activity of many units causes those effects. It is unnecessary to have a separate higher-level unit that detects the pattern of activity and causes the appropriate effects.

1.2. SYSTEMS OF SIMPLE UNITS WITH MODIFIABLE INTERCONNECTIONS

This section describes some models in which changes in the strengths of the interconnections in a system of simple units are used to implement category formation and associative memory. Before introducing these models, however, we outline the ideas about “formal” neurons that were largely responsible for the choice of the particular kind of simple unit used in these models.

1.2.1. The McCulloch-Pitts Neuron

Probably the best known, and arguably the most influential model of the nervous system, even today, is that proposed in 1943 by Warren McCulloch and Walter Pitts. They approximated the brain as a set of binary elements—abstract neurons which were either on or off—that realized the statements of formal logic. To quote the first sentence of the abstract of their paper (McCulloch & Pitts, 1943):

Because of the “all-or-none” character of nervous activity, neural events and the relations between them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms … and that for any logical expression satisfying certain conditions, one can find a net behaving in the fashion it describes [p. 115].

One finds in their paper much of the machinery familiar to those who study automata theory: binary elements, threshold logic, and quantized time where the state of the system at the (n + 1)th moment reflects the states of the inputs to the elements at the nth moment. The primary result of their paper was that nets of such neurons were perfectly general in that they could realize any finite logical expression.

This model obviously has practical implications: put together such neurons and you can make a powerful, general computing device. At about the time of the 1943 paper, exactly such a project was underway at the Moore School of Engineering of the University of Pennsylvania. This paper on brain modeling had an influence on John von Neumann when he sketched the logical outline of the first modern digital computer—the first machine with a program stored with the data.

In a famous technical report, von Neumann (1945) said:

Every digital computing device contains certain relay like elements with discrete equilibria. Such an element has two or more distinct states in which it can exist indefinitely…. The relay action manifests itself in the emission of stimuli by the element whenever it has itself received a stimulus of the type indicated…. It is worth mentioning that the neurons of the higher animals are definitely elements in the above sense…. Following W. Pitts and W. S. McCulloch … we ignore the more complicated aspects of neuron functioning … [p. 360].

1.2.2. Perceptrons

The perceptron, originally developed by Rosenblatt, and related models such as MADALINE and ADALINE developed by Widrow were intensively studied in the early 1960s. These models have now become part of the lore of pattern recognition, and good short introductions are available in many books on pattern recognition, as well as in the classic books, Learning Ma...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Introduction to the Updated Edition
• Introduction
• 1. Models of Information Processing in the Brain
• 2. A Connectionist Model of Visual Memory
• 3. Holography, Associative Memory, and Inductive Generalization
• 4. Storage and Processing of Information in Distributed Associative Memory Systems
• 5. Representing Implicit Knowledge
• 6. Implementing Semantic Networks in Parallel Hardware
• 7. Skeleton Filters in the Brain
• 8. Categorization and Selective Neurons
• 9. Notes on a Self-organizing Machine
• 10. Parallel-Processing Mechanisms and Processing of Organized Information in Human Memory
• Author Index
• Subject Index