Connectionist-Symbolic Integration

From Unified to Hybrid Approaches

Ron Sun, Frederic Alexandre

• Publisher: Psychology Press
• Year: 2013
• Print ISBN:
  9780805823486
• eBook ISBN:
  9781134802135
• Topic: Psychology
• Subtopic: Cognitive Psychology & Cognition
• Index: Psychology

A variety of ideas, approaches, and techniques exist -- in terms of both architecture and learning -- and this abundance seems to lead to many exciting possibilities in terms of theoretical advances and application potentials. Despite the apparent diversity, there is clearly an underlying unifying theme: architectures that bring together symbolic and connectionist models to achieve a synthesis and synergy of the two different paradigms, and the learning and knowledge acquisition methods for developing such architectures. More effort needs to be extended to exploit the possibilities and opportunities in this area.

This book is the outgrowth of The IJCAI Workshop on Connectionist-Symbolic Integration: From Unified to Hybrid Approaches, held in conjunction with the fourteenth International Joint Conference on Artificial Intelligence (IJCAI '95). Featuring various presentations and discussions, this two-day workshop brought to light many new ideas, controversies, and syntheses which lead to the present volume.

This book is concerned with the development, analysis, and application of hybrid connectionist-symbolic models in artificial intelligence and cognitive science. Drawing contributions from a large international group of experts, it describes and compares a variety of models in this area. The types of models discussed cover a wide range of the evolving spectrum of hybrid models, thus serving as a well-balanced progress report on the state of the art. As such, this volume provides an information clearinghouse for various proposed approaches and models that share the common belief that connectionist and symbolic models can be usefully combined and integrated, and such integration may lead to significant advances in understanding intelligence.

1

AN INTRODUCTION TO HYBRID CONNECTIONIST-SYMBOLIC MODELS

Ron Sun

Department of Computer Science
The University of Alabama

1 MOTIVATIONS

There has been a considerable amount of research in integrating connectionist and symbolic processing. While such an approach has clear advantages, it also encounters serious difficulties and challenges. Consequently, various ideas and models have been proposed to address different problems and different aspects in this integration. The need for such models has been slowly but steadily growing over the past five years, from many segments of the artificial intelligence and cognitive science communities, ranging from expert systems to cognitive modeling and to logical reasoning. Some interesting and important approaches have been developed. There has been a general consensus that hybrid connectionist-symbolic models constitute a promising avenue toward developing more robust, more powerful, and more versatile architectures, both for cognitive modeling and for intelligent systems. It is definitely worthwhile pursuing research in this area further still, which might generate important new ideas and significant new applications.

The basic motivations for research in hybrid connectionist-symbolic models can be briefly summarized as follows:

■ Cognitive processes are not homogeneous; a wide variety of representations and mechanisms are employed. Some parts of cognitive processes are best captured by symbolic models, while others by connectionist models (Smolensky 1988, Sun 1995). Therefore, a need for “pluralism” exists in cognitive modeling, which leads to the development of hybrid models as tools and frameworks.

■ The development of intelligent systems for practical applications can benefit greatly from a proper combination of different techniques, since no one single technique can do everything, as is the case in many application domains, ranging from bank loan approval to industrial process control (Medsker 1994). By combining different techniques, intelligent systems can explore the synergy of these techniques.

■ To develop a full range of capabilities in autonomous agents, an autonomous agent architecture needs to incorporate both symbolic and subsymbolic processing for handling declarative and procedural knowledge, respectively, in order to effectively deal with a variety of environments in which an agent finds itself (Sun and Peterson 1995). Such an agent architecture, incorporating both conceptual and subconceptual processes, leads naturally to a combination of symbolic models (which capture conceptual processes) and connectionist models (which capture subconceptual processes).

The book tries to bring to light many new ideas, controversies, and syntheses in this broad area. The focus is on learning and architectures that feature hybrid representations and support hybrid learning.

2 IMPORTANT ISSUES

There have been many important and/or crucial issues that have been raised with regard to hybrid connectionist-symbolic models. These issues concern architectures of these models, learning in these models, and various other aspects.

Hybrid models involve a variety of different types of processes and representations, in both learning and performance. Therefore, multiple mechanisms interact in complex ways in most of these models. We need to consider seriously ways of structuring these different components; in other words, we need to consider architectures, which thus occupy a clearly more prominent place in this area of research compared with other areas in AI. Some architecture-related issues are as follows:

■ What type of architecture facilitates what type of process?

■ Should hybrid architectures be modular or monolithic?

■ For modular architectures, should we use different representations in different modules of an architecture or should we use the same representation throughout?

■ How do we decide if a particular part of an architecture should be symbolic, localist, or distributed in its representation?

■ How do we structure different representations in different parts to achieve optimal results?

■ How do we incorporate prior knowledge into hybrid architectures?

Although purely connectionist models, which constitute a part of any hybrid model, are known to excel in their learning abilities, hybridization makes it more difficult to do learning. Most symbolic models and architectures are not specifically designed to perform learning, especially not in a fully autonomous and bottom-up fashion, and most of them have difficulties with learning in some ways. Therefore, the hybridization of connectionist and symbolic models inherits the difficulty of learning from the symbolic side and mitigates to some large extent the advantage that the purely connectionist models have in their learning abilities. Considering the importance of learning in both modeling cognition and building intelligent systems, it is crucial for researchers in this area to pay more attention to ways of enhancing hybrid models in this regard and to putting learning back into hybrid models. Some of the learning-related issues that need to be addressed include:

■ How can learning be incorporated and utilized in each type of architecture?

■ What kinds of learning can be done in each type of architecture, respectively?

■ How do learning and representation interact along the developmental line?

■ What is the relationship between symbolic machine learning methods, knowledge acquisition methods, and connectionist (neural network) learning algorithms, especially in the context of hybrid models?

■ How can each type of architecture be developed with various combinations of the above-mentioned methods?

■ How can learning algorithms be developed for (usually knowledge-based) localist connectionist networks?

Figure 1 Classifications of Hybrid Models

■ How can rules be extracted from, and refined by, (hybrid) connectionist models?

■ How can complex symbolic structures besides rules, such as frames and semantic networks, be learned in hybrid connectionist models?

3 ARCHITECTURES

In terms of architectures of hybrid models, various distinctions, divisions, and classifications have been proposed and discussed. (see chapter 2). As a first cut, we can divide these models up into two broad categories: single-module architectures and multi-module architectures (including both homogeneous and heterogeneous multi-module architectures). See Figure 1.

For single-module architectures, along the representation dimension, there can be the following types of representations (see Sun and Bookman 1994): symbolic (as in conventional symbolic models, in which case, the model is no longer a hybrid model), localist (with one distinct node for representing each concept; for example, Lange and Dyer 1989, Sun 1992, Shastri and Ajjanagadde 1993, Barnden 1994), and distributed (with a set of non-exclusive, overlapping nodes for representing each concept; for example, Pollack 1990, Sharkey 1991). Usually, it is easier to incorporate prior knowledge into localist models since their structures can be made to directly correspond to that of symbolic knowledge (Fu 1991). On the other hand, connectionist learning usually leads to distributed representation, such as in the case of backpropagation learning. Along a different dimension, in terms of mappings between symbolic and connectionist structures (Hilario 1995, Medsker 1994), we see that there are the direct translational approach, which creates a network structure that directly corresponds to the symbolic structure to be implemented (usually in a localist network), such as in the implementation of rules in a backpropagation network by Fu (1991) and Towell and Shavlik (1993), and the transformational approach, which creates the equivalent of symbolic structures in connectionist networks without actually embedding the structures directly in networks, such as the encoding of trees in RAAM (Pollack 1990). The relative advantage of each is a still unsettled issue (which is related to the compositionality issue as being debated in the theoretical community). Another possible dimension is in terms of the dynamics of the models, rather than in terms of the static topology (i.e., the static mapping) of the networks used; that is, we can classsify models based on whether their internal dynamics is translational or transformational, which can be highly correlated with but not necessarily identical to the static topology of networks.

For multi-module models, we can distinguish between homogeneous models and heterogeneous models. Homogeneous models may be very much like a single-module model discussed above, except they contain several replicated copies of the same underlying structure, each of which can be used for processing the same set of inputs, to provide redundancy for various reasons. For example, we can have competing experts (of the same domain), each of which may vote for a particular solution. Or, each module (of the same makeup) can be specialized (content-wise) for processing a particular type of input or another; for example, we can have different experts with the same structure and representation but different content/knowledge for dealing with different situations.

For heterogeneous multi-module models, a variety of distinctions can be made. First of all, a distinction can be made in terms of representations of constituent modules. In multi-module models, there can be different combinations of different types of constituent modules: for example, a model can be a combination of localist and distributed modules (for example, CONSYDERR as described in Sun 1995, for cognitive modeling of commonsense reasoning and decision making), or it can be a combination of symbolic modules and connectionist modules (either localist or distributed; for example, SCRUFFY as described in Hendler 1991, mainly for practical applications).

Another distinction that can be made is in terms of the coupling of modules: a set of modules can be either loosely coupled or tightly coupled (Medsker 1994). In loosely coupled situations, modules communicate with each other, primarily through some interfaces as in, for example, SCRUFFY (Hendler 1991). Such loose coupling enables some loose forms of cooperation among modules. One form of cooperation is in terms of pre/postprocessing vs. main processing: while one or more modules take care of pre/postprocessing, such as transforming input data or rectifying output data, a main module focuses on the main part of the processing task. This is probably the simplest and earliest form for hybrid systems, in. which, commonly, pre/post processing is done using a connectionist network and the main task is accomplished through the use of symbolic methods (as in conventional expert systems). Another form of cooperation is through master-slave relationships: while one module maintains control of the task at hand, it can call upon other modules to handle some specific aspects of the task. For example, a symbolic expert system, as part of a rule, may invoke a neural network to perform a specific classification or decision making or some other processing. A variation of this form is the processor-monitor (meta-processor) combination, in which a processing module does the work while a monitor module waits for certain events to occur in which case the monitor will inform and/or alter the working of the processing module. Yet another form of cooperation is the equal partnership of multiple modules. In this form, the modules (the equal partners) can consist of (1) complementary processes, such as in the SOAR/ECHO combination (see chapter 6), or (2) multiple functionally equivalent but representationally different processes, such as in the CLARION architecture (chapter 7), or (3) they may consist of multiple differentially specialized and heterogeneously represented experts each of which constitutes an equal partner in accomplishing the task.¹

In tightly coupled systems, on the other hand, the constituent modules interact through multiple channels or may even have node-to-node connections across two modules, such as CONSYDERR (Sun 1995) in which each node in one module is connected to a corresponding node in the other module. For tightly coupled multi-module systems, there are also a variety of different forms of cooperation among modules, in ways quite similar to loosely coupled systems. Such forms include master-slave, processor-monitor, and equal partnership, each of which is basically the same as in loosely coupled systems, except in this case a larger number of connections exist and a lot more interactions are occuring among modules. However, another possibility in loosely coupled systems, i.e., pre/post-processing, is not one of the possibilities with tightly coupled systems, since it entails loose connections between the pre/post-processing module and the main processing modules.

Another distinction that can be made of all multi-module systems is with regard to the granularity of modules in such systems: they can be coarse-grained or fine-grained. On one end of the spectrum, a multi-module system can be very coarse-grained so ...

Table of contents

• Cover
• Halftitle
• Title
• Copyright
• Contents
• Preface
• 1 An Introduction to Hybrid Connectionist-Symbolic Models
• Part I Reviews and Overviews
• Part II Learning in Multi-Module Systems
• Part III Representing Symbolic Knowledge
• Part IV Acquiring Distributed Representation
• Part V Epilog
• About the Editors
• Index