Learning and Categorization in Modular Neural Networks

Jacob M.J. Murre

• Publisher: Psychology Press
• Year: 2014
• ISBN:
  9781317781363
• Edition: 1
• Topic: Psychologie
• Subtopic: Histoire et théorie en psychologie

This book introduces a new neural network model called CALM, for categorization and learning in neural networks. The author demonstrates how this model can learn the word superiority effect for letter recognition, and discusses a series of studies that simulate experiments in implicit and explicit memory, involving normal and amnesic patients. Pathological, but psychologically accurate, behavior is produced by "lesioning" the arousal system of these models. A concise introduction to genetic algorithms, a new computing method based on the biological metaphor of evolution, and a demonstration on how these algorithms can design network architectures with superior performance are included in this volume. The role of modularity in parallel hardware and software implementations is considered, including transputer networks and a dedicated 400-processor neurocomputer built by the developers of CALM in cooperation with Delft Technical University. Concluding with an evaluation of the psychological and biological plausibility of CALM models, the book offers a general discussion of catastrophic interference, generalization, and representational capacity of modular neural networks. Researchers in cognitive science, neuroscience, computer simulation sciences, parallel computer architectures, and pattern recognition will be interested in this volume, as well as anyone engaged in the study of neural networks, neurocomputers, and neurosimulators.

PART I

CALM: Categorizing And Learning Module

Chapter 1

Introduction

1.1 The importance of learning

The study of learning systems is important, both from a theoretical and from a practical point of view. Theoretically, the study of the ability of information processing systems to learn may result in better models for understanding how living organisms adapt to environmental demands by selecting and storing information. From a practical point of view, it may solve some programming problems encountered in the development of complex, intelligent information processing systems. Because a learning system may show autonomous programming, it can develop its own structure and function in interaction with the environment, without the need for detailed instructions by a human controller.

The recent interest in connectionism (Feldman, 1981; Feldman and Ballard, 1982), parallel distributed processing (Rumelhart and McClelland, 1986; McClelland and Rumelhart, 1986a), neural networks (e.g. Hinton and Anderson, 1981; Grossberg, 1982, 1987a; Kohonen, 1989a), and neurocomputing (Anderson and Rosenfeld, 1988) has given a new impulse to the study of learning in information processing models. The quasi-neural elements and syntax of the connectionist language provide a useful formalism that is very well suited for incorporating learning abilities without, however, restricting the field to the extent that it could be called a unitary theory. Just like their biological example, some learning networks may self-organize by selecting and categorizing relevant information, and by retaining this over time. It is hoped that by merely placing a learning network in an environment implicit ‘programs’ will emerge as a consequence of the interaction of the network with this environment.

Although the importance of learning has been recognized in the behavioural sciences for a long time (e.g. Ebbinghaus, 1885; James, 1890), it has been relatively neglected in conventional cognitive models and classical artificial intelligence systems, based on the computer metaphor. These models are often based on the assumption that it is possible to define and to formalize all necessary knowledge, and to incorporate this knowledge in a model. Such a view, of course, limits the knowledge that can be represented in an information processing model to the knowledge that can be made explicit by the designer. Moreover, the presupposition that all knowledge can be implemented in the form of well-defined rules is neither proven nor probable (see Wolters and Phaf, 1990). Pattern recognition, for example, which requires the simultaneous processing and integration of a large amount of information, is an ability that the human system can perform with ease, but has proven an exceedingly hard task for most conventional, rule-based approaches. Even at this stage of development, learning neural networks perform functions for which no computing algorithm has been found. The robot arm developed by Kuperstein (1988), for instance, learns to direct its arm to objects detected by its two video cameras, a task that so far has defied a rigorous analytic solution.

The idea of non-formalizable knowledge that can be learned by, but not programmed into, an information processing system may have profound consequences for the practice of connectionism. Approaches to neural networks, that want to describe the knowledge represented in a network in a mathematical fashion, may restrict their models to functions that bear little relevance to the functions that characterize the natural systems. If easily formalizable functions were to exist, a hundred years of experimental behavioural research would probably have yielded such general functions. In fact, it can be argued that, when a model can be completely formulated in the mathematical language, a connectionist formulation is not necessary. Although mathematical analysis may be an important goal for connectionist research, we adhere to the standpoint that psychological plausibility of neural network models should take precedence over rigorous mathematical tractability. In this work a new learning network model will be presented starting from some more psychologically and biologically oriented considerations.

1.2 Some problems with learning neural networks

Approaches to learning information processing systems are still in their infancy, and despite preliminary successes, most of the currently popular learning network models show a number of shortcomings and problems. Among the problems are lack of speed, lack of stability, inability to learn either with or without supervision, and inability both to discriminate between and generalize over patterns.

A lack of speed, in particular, is hampering completely structureless, homogeneous models and models that assume a hierarchical, layered structure with total connectivity between nodes of adjacent levels. If the size of the models increases, the quadratic rise in the number of modifiable connections may lead to a prohibitive lengthening of the time to reach stable states (e.g. Perugini and Engeler, 1989). The network by Rumelhart et al. (1986), for instance, already needs hundreds to thousands of presentations to learn even a simple function such as the EXOR. Though many improvements of the backpropagation procedure have been proposed, lack of speed still remains a major problem for this class of networks.

Total homogeneous connectivity may also result in a lack of stability of representations, because there is too much susceptibility to interference. Every input-output relation that can be learned by such a network has an a priori equal status to all other possible relations. Every new relation can and will, therefore, interfere with every old relation, if the latter is not strengthened over and over. The problem of ‘catastrophic interference’ in neural networks has been discussed by McCloskey and Cohen (1989) and by Ratcliff (1990). Much recent work is aimed at investigating this problem (see Chapter 7 for an overview). They performed extensive tests on the backpropagation procedure and showed that well-learned information is replaced rapidly when new information is learned and the old information is not presented again. If such a network has learned a stimulus set A to perfection and is subsequently trained on another set B, it may be able to learn the set B to perfection as well, but in the course of training it will forget most of set A. One way to accomplish perfect learning of both sets is to retrain over and over with both A and B. Several extensions of backpropagation have recently been proposed to deal with this issue. It appears that full connectivity is only one of the problems causing massive interference (see Chapter 7 for a more detailed discussion of this problem).

Another problem with the delta-rule and backpropagation learning procedures is that they only allow for supervised, but not unsupervised, learning. When these networks are said to function without supervision, usually an autosupervision scheme is used, where input and desired output pattern are the same. Of course, from a psychological point of view, supervised learning (i.e. with instruction and correction of errors) may also be important. Indeed, even in many cases without apparent explicit supervision, such as uninstructed skill learning and operant conditioning, there may still be autosupervision. The result of some action may be compared with some internal standard and the action may be corrected accordingly. Yet, much learning, like the incidental storage of everyday experiences, proceeds without any form of supervision and the inability of these learning procedures to handle such learning seems to be a severe shortcoming.

As far as unsupervised learning is accomplished by using some form of Hebbian-type learning rule most models are capable of learning by autonomously discriminating between different input patterns. Some others allow for generalization over similar input patterns as may be necessary for the recognition of constancies, like invariance for size, and translation as, for instance, in handwriting. However, none of the existing models seems to combine both abilities in an efficient manner. For example, the well-known adaptive resonance theory (ART) (Carpenter and Grossberg, 1987; Grossberg, 1976) is capable of further and further discriminations, but it may have only a limited capacity to cluster patterns under a common representation. Patterns that do not sufficiently match any of the learned representations may be rejected by an ART module that is completely ‘filled’ with other representations.

So, it seems that most currently available learning networks have architectural characteristics, or use learning rules, that result in a variety of problems and shortcomings in comparison with the human system. In part, this may be caused by the fact that neural network research seems to have been guided primarily by the availability of computational and mathematical methods, rather than by biological or psychological constraints. It will be argued here that, if the current connectionist language is supplemented with new terms derived from psychology and the neurosciences, the combined effort may lead to more plausible network models and may help to solve some of the above problems.

1.3 Structural constraints

1.3.1 Limitations on connectivity: modularity

Many of the currently popular network architectures show little, if any, structural constraints. Some networks assume total interconnectivity between all nodes (e.g. Hopfield, 1982). Others assume a hierarchical, multi-layered structure (e.g. Rumelhart et al., 1986) in which each node in a layer is connected to all nodes in neighbouring layers.

Completely connected architectures allow virtually any possible input–output relation (Funahashi, 1989; Hornik et al., 1989; Stinchcombe and White, 1989) to be learned. They exhibit extreme plasticity. The position taken by these connectionists in the nature-nurture debate seems to be at the far ‘nurture’ end. Psychological evidence, however, indicates that for the human system such an extreme position may not be warranted. Some learning tasks seem to be much easier than others (e.g. learning a motor skill with the preferred or the non-preferred hand). Complete connectivity also suggests that any multi-task execution would be hampered by mutual interference. But results from interference studies in humans show that many tasks can be performed simultaneously without mutual interference (e.g. speaking while driving a car, or perceiving and producing speech; Shallice et al., 1985), whereas other task combinations are almost impossible to perform at the same time, especially if some task elements are shared (e.g. listening to two conversations at the same time (see also Allport, 1980)). Furthermore, there is a large body of neuropsychological evidence showing that isolated abilities, such as the ability to recognize faces (e.g. Damasio et al., 1982), or to speak fluently, may be lost without affecting other cognitive abilities in any way (e.g. Gazzaniga, 1989; Luria, 1973; Shallice, 1988). In summary, these data seem to indicate that the human information processing system consists of modules – relatively isolated subsystems – that can function quite independently of each other.

Also, neuroanatomy provides a wealth of evidence showing that the human brain does not have total and uniform connectivity. For one thing, this would require 10¹¹ connections per neuron, whereas only about 10⁴ connections are available. On a macroscopic scale many structurally different cortical and sub-cortical centres can be distinguished that are only partly interconnected (Kosslyn et al., 1990; Livingstone and Hubel, 1988; Zeki and Shipp, 1988). On a microscopic level, the minicolumns found in the grey matter of the neocortex can be considered module-like structures (e.g. Mountcastle, 1978; Szentágothai, 1975). The minicolumns consist of small regions in the neocortex having intracolumnar connections that are for a large part inhibitory, and long-range afferent and efferent excitatory connections to other neocortical columns via t...