Philosophy and Connectionist Theory

William Ramsey, David E. Rumelhart, Stephen P. Stich

• Publisher: Psychology Press
• Year: 2013
• Topic: Psychology
• eBook ISBN:
  9781134755097
• Subtopic: History & Theory in Psychology
• Index: Psychology

The philosophy of cognitive science has recently become one of the most exciting and fastest growing domains of philosophical inquiry and analysis. Until the early 1980s, nearly all of the models developed treated cognitive processes -- like problem solving, language comprehension, memory, and higher visual processing -- as rule-governed symbol manipulation. However, this situation has changed dramatically over the last half dozen years. In that period there has been an enormous shift of attention toward connectionist models of cognition that are inspired by the network-like architecture of the brain. Because of their unique architecture and style of processing, connectionist systems are generally regarded as radically different from the more traditional symbol manipulation models.

This collection was designed to provide philosophers who have been working in the area of cognitive science with a forum for expressing their views on these recent developments. Because the symbol-manipulating paradigm has been so important to the work of contemporary philosophers, many have watched the emergence of connectionism with considerable interest. The contributors take very different stands toward connectionism, but all agree that the potential exists for a radical shift in the way many philosophers think of various aspects of cognition. Exploring this potential and other philosophical dimensions of connectionist research is the aim of this volume.

II REPRESENTATION IN CONNECTIONIST MODELS

3	What is the “D” in “PDP”? A Survey of the Concept of Distribution

Tim van Gelder

Indiana University

Suppose there were such a thing as the “computational theory of mind” (CTM); and suppose that, for whatever reason, you were dissatisfied with it. You may well be tempted to ask: What would an alternative look like? Could there be an alternative that was even remotely plausible? Is connectionism in the business of developing such an alternative?

With issues such as these in vogue recently, considerable attention has been given to the preparatory task of succinctly characterizing some version of CTM to which the desired alternative can stand opposed. One point of universal consensus has been that an essential feature of CTM is the use of symbolic representations. Any theory failing to employ such representations automatically falls outside the broad CTM umbrella. This suggests an obvious approach to the questions just raised. Assuming that any remotely plausible theory of mind must be based on manipulation of internal representations of some kind, we need to find some other generic form of representation to play a foundational role in the new theory analogous to that played by symbolic representation in CTM. Having found such a form, we could evaluate the general plausibility of a theory of mind constructed around it. Perhaps connectionist work contains some clues here, both about the form itself and about the kind of theory in which it would be embedded.

The alternative form of representation required by this approach has to satisfy some demanding conditions. It must, of course, be demonstrably nonsymbolic, but it must also be sufficiently general to allow the characterization of a reasonably broad conception of the mind. This means, among other things, that it must be rich enough to encompass a wide variety of particular articulations (just as symbolic representation can be instantiated in a very wide variety of particular ways), and yet characterizable in a way that is sufficiently abstract to transcend all kinds of irrelevant implementation details. Crucially, it will have to be powerful enough to make possible the effective representing of the kinds of information that are essential to human cognitive performance. Preferably, this alternative will have some deep connection with neural network architectures, thereby minimizing future difficulties relating the new theory to the neurobiological details, and in the meantime allowing us to both interpret and learn from connectionist research.

This is a tall order by any account, and a moment’s reflection reveals that there are few if any plausible candidates available. The traditional cognitive science literature is of remarkably little help here. In that relatively small portion concerned specifically with the actual form of mental representation, symbolic styles have for the most part been contrasted only with broadly imagistic styles (pictorial, analog, etc.). For a long period the most notable research taking a manifestly nonsymbolic approach was the investigation of mental imagery, and surveys of the field typically treat these two broad categories as the only relevant possibilities. Yet, although the category of imagistic representations might begin to satisfy some of the constraints just listed, it clearly fails to satisfy others; in particular, it is generally accepted that imagistic representations are not powerful enough to underlie central aspects of cognition such as linguistic performance and problem solving. There has even been serious debate over whether mental imagery itself is strictly imagistic.

One response to this apparent lack of plausible alternatives is to accept that representations must be symbolic in some suitably generic sense, and consequently to maintain that any feasible alternative to CTM must differ not in how knowledge is represented but rather in how the representations themselves are manipulated—that is, in the nature of the mental processes. Yet this approach also is unpromising. Representations and processes tend to go hand in hand; the way knowledge is represented largely fixes appropriate processes and vice versa. For this reason, conceding that representations must be generically symbolic places one in a conceptual vortex with the standard CTM at the center.

One reason there appear to be so few alternatives is that the conception of symbolic representation invoked in characterizations of CTM is so very general, and usually rather vague. This suggests a more cautious gambit: fine-tune the conception of symbolic representation itself, articulating some more specific formulation that can fairly be attributed to CTM, thereby making room for some quasi-symbolic alternative between analog anarchy on one hand and the rigors of strictly syntactic structure on the other. However, although headway can certainly be made in this direction, it has an obvious strategic flaw: Major differences in paradigms are unlikely to rest on delicate philosophical distinctions, and if perchance they did, it would be relatively difficult to convince others of the fact. It is vastly preferable to propose a style of representation with unquestionable antisymbolic credentials. If there actually is any quasi-symbolic option of the kind just mentioned, it should be introduced as a special case of a manifestly distinct category, rather than as some subtle variant on the standard symbolic model.

If at this point we look to connectionism, it is difficult to avoid noticing the frequent emphasis on distributed representation, an emphasis evident even in the familiar designation “parallel distributed processing” (PDP). Distributed representation may well satisfy the first of the requirements on an acceptable alternative, because it is often deliberately contrasted with symbolic representation (e.g., as when it is claimed that a network knows how to form the past tense without the benefit of explicit symbolic rules). Moreover, the category appears to be appropriately general; at one time or another, distributed representation has cropped up in areas as diverse as functional neuroanatomy, psychology of memory, image processing, and optical phenomena such as holography; indeed, researchers originally began applying the term distributed representation in connectionist contexts precisely because of perceived similarities between connectionist representations and these other cases. Considerations such as these suggest we should inquire into the possibility that distributed representations form the kind of category we are after. Perhaps, in other words, there is here a natural kind of representation, a kind that includes all or most of the cases previously described as distributed, whose members are somehow inherently nonsymbolic, but that is nevertheless sufficiently rich, powerful, and so on, that it might form the basis of some plausible alternative to CTM.

The immediate difficulty with this suggestion is the lack of any clear account of what distributed representation actually is. The concept itself is relatively novel, and though many people have recently offered their preferred brief characterizations, it has had almost no serious treatments as an independent topic of investigation.¹ Worse, there is very little consensus even in such characterizations as are available. The diversity of definitions suggests that there really is no unified category of distributed representations after all. Feldman (1989) for one has concluded “…people have been using the term [“distributed”] to denote everything from a fully holographic model to one where two units help code a concept; thus, the term has lost its usefulness (p. 72). Clearly, before we can even begin to take seriously the idea that a plausible alternative to CTM might be constructed on a distributed foundation, we need to formulate a reasonably clear and comprehensive account of the nature of distributed representation. This task goes vastly beyond what might be achieved here; what follows is simply an exploratory overview of the current concept (or concepts), a disentangling of some of the many themes and issues that have at one time or another been associated with distribution.

DIVERSE DEFINITIONS OF DISTRIBUTION

A useful point of entry is to note the inadequacy of one style of definition common in connectionist work. In perhaps the most authoritative version (Hinton, McClelland, & Rumelhart, 1986), representations are alleged to be distributed if: “Each entity is represented by a pattern of activity distributed over many computing elements, and each computing element is involved in representing many different entities” (p. 77).² The most obvious problem here, from the current perspective, is one of narrow focus. A distributed representation is defined as a “pattern of activity" over “computing elements” specifically; but this is too limited even for connectionist purposes, because there at least two species of distributed representation in connectionist networks—the patterns of activity themselves, and the patterns of connectivity that mediate their transformation. It may be that these are in fact essentially interlocked, each needing the other, but there is at least a prima facie distinction, because the two kinds of representation appear to have some significantly different characteristics. Thus this definition would have to be generalized significantly if it were to capture the notion of distribution implicit even in connectionist work, let alone whatever is common to cases as diverse as those mentioned above.

Narrowness is not however the worst of its problems. The intended contrast is with a variety of “localist” representation in which each entity is represented by activity in a single computing element. But in its concern to distinguish distribution from these kinds of localist cases, this definition patently fails to distinguish it from other cases that, surely, are not distributed in any interesting sense. For many familiar kinds of representation count as “patterns of activity” over sets of “computing elements” (“units,” “locations,” or whatever); in particular, when numbers are encoded as strings of bits in a register of an ordinary pocket calculator, they are being represented by distinctive activity patterns, and each unit or location participates in the representing of many different numbers over the course of a calculation.³ This leaves two possibilities: either distribution is not an interestingly distinct category after all; or it is, but one whose essence eludes this definition. The latter turns out to be vastly more fertile as a working hypothesis.

Day to day practice often compensates for deficiencies in overt formulation. The real content of this characterization is implicit in the way it is received and guides construction of new connectionist schemes of representation. In this light, the central theme of this version—the representing of entities as different patterns of activity over groups of units—deserves closer scrutiny. Consider first the very simple requirement that entities be represented over many units—or, more generally, over some relatively extended portion of the resources available for representing.⁴ Lacking any better term, I will describe representations that are spread out in this sense as extended. Clearly, a representation can only be extended by comparison with some normal, minimum or standard form, which can vary from case to case and style to style. Thus in typical connectionist networks the benchmark is one computing unit to every item, and relative to this a representation is extended if it uses many units for every item. In the brain the most plausible minimal unit is presumably the neuron (as in “grandmother” or “yellow Volkswagen” cells). Note however that in some other cases, such as optical holography—generally taken to be a paradigm example of distributed representation—there is no obvious parallel, because the surface of a photographic plate does not come naturally partitioned.

This bare notion of extendedness may seem trivial, but it is a very common theme in characterizations of distribution; indeed, on some occasions distribution is described solely in such terms.⁵ It is therefore interesting to see what, if anything, is gained by distributing even in this minimal sense.

An important practical concern is worth mentioning first: extendedness can buy a certain kind of reliability or robustness. If an item is represented over many locations in such a way that no particular location is crucial to overall efficacy, then the system can withstand small and isolated damage or noise relatively well. This point is illustrated by the benefits of redundancy. Duplicating a given representation many times obviously increases the ability of the whole collection to convey the same content under adverse conditions. Thus, one reason for the industrious copying of medieval manuscripts was to ensure that if any one were lost, the same text would be preserved elsewhere—a point with a modern counterpart for users of word processors. An extended representation need not be simply redundant, however. Instead of activating a single neuron to represent a given perceptual item, the brain activates a vast number, forming an overall pattern for the same purpose, but where each neuron is tuned in a slightly different way to the retinal input. Loss of any particular neuron, or noise in the system, has almost no effect on the overall effectiveness of this representation, which is fortunate, given how noisy neurons are and the rate at which we lose them.

Whether this advantage of noise or damage resistance in fact accrues to a given case of extended representation depends very much on the form of encoding involved. A particular number N might be represented in a digital computer either in binary form or, in a more extended fashion, as a string of bits of length N; neither has any particular advantage of reliability over the other, even though the latter uses vastly more resources. This is just to stress again the point that extendedness must be achieved in such a way that no particular unit or location is crucial, a condition violated in both these cases. Further, the portion of the resources involved should be large not only relative to some theoretical minimum (such as the unit, neuron, or location), but also relative to the scale of likely damage or noise in the system itself.

Whether a representation is extended is independent of the stronger requirement that a given representation take the form of a distinctive pattern over that larger portion of the resources. Despite this independence, an important advantage in distributing representations in this sense is that it makes possible the use of a distinctive pattern for each distinct item. Indeed, such an approach will be essential if we want to represent a number of different items over the same set of units. Many authors, especially connectionists and commentators on connectionism, claim that the essence of distribution is to be found in this shift to the level of overall patterns, or, more generally, to characteristic overall states of the network or system. Rosenfeld and Touretzky (1988), for example, have defined schemes of distributed representation as those in which “each entity is represented by a pattern of activity over many units (p. 463).⁶

These patterns might be completely unrelated; they might be chosen at random, or it might suit one’s computational purposes to choose patterns simply so as to maximize the distinctness of any two representations. On the other hand, an important reason for moving to characteristic patterns for the representing of each item is that the internal structures of these patterns can be systematically related, both to each other and to the nature of the items to be represented, thereby making the overall scheme more useful in certain ways. There are many ways to develop pattern-based schemes of representation in which the internal structures of the patterns have this kind of systematic semantic significance, but one in particular has been especially popular in connectionism. On this approach individual processing units pick out (micro)features, which are simply aspects of the domain, though usually at a much finer grain than that of the primary items to be represented. In a well known example, in order to represent a kind of room, we first assign to individual units features that are typically found in various kinds of rooms, such as sofa, TV, ceiling and stove.⁷ Each different kind of room can then be represented by means of a distinctive pattern over these units—that is, that pattern that picks out all and only the relevant features. In this way it is possible to generate patterns for representing items in which semantic differences are built directly into the internal structure.

The popularity of this approach in connectionism ha...

Table of contents

• Cover
• Halftitle
• Title
• Copyright
• Contents
• List of Contributors
• Series Foreword
• Preface
• I Connectionism and Other Styles of Cognitive Modeling
• II Representation in Connectionist Models
• III Philosophical Implications of Connectionism
• Author Index
• Subject Index