Artificial Intelligence in the Age of Neural Networks and Brain Computing

Robert Kozma, Cesare Alippi, Yoonsuck Choe, Francesco Carlo Morabito

• Publisher: Academic Press
• Year: 2018
• eBook ISBN: 9780128162507
• Topic: Biological Sciences
• Subtopic: Biotechnology
• Index: Biological Sciences

Artificial Intelligence in the Age of Neural Networks and Brain Computing demonstrates that existing disruptive implications and applications of AI is a development of the unique attributes of neural networks, mainly machine learning, distributed architectures, massive parallel processing, black-box inference, intrinsic nonlinearity and smart autonomous search engines. The book covers the major basic ideas of brain-like computing behind AI, provides a framework to deep learning, and launches novel and intriguing paradigms as future alternatives. The success of AI-based commercial products proposed by top industry leaders, such as Google, IBM, Microsoft, Intel and Amazon can be interpreted using this book.- Developed from the 30th anniversary of the International Neural Network Society (INNS) and the 2017 International Joint Conference on Neural Networks (IJCNN)- Authored by top experts, global field pioneers and researchers working on cutting-edge applications in signal processing, speech recognition, games, adaptive control and decision-making- Edited by high-level academics and researchers in intelligent systems and neural networks

Chapter 1

Nature's Learning Rule

The Hebbian-LMS Algorithm

Bernard Widrow, Youngsik Kim, Dookun Park, and Jose Krause Perin Department of Electrical Engineering, Stanford University, Stanford, CA, United States

Abstract

Hebbian learning is widely accepted in the fields of psychology, neurology, and neurobiology. It is one of the fundamental premises of neuroscience. The LMS (least mean square) algorithm of Widrow and Hoff is the world's most widely used adaptive algorithm, fundamental in the fields of signal processing, control systems, communication systems, pattern recognition, and artificial neural networks. These learning paradigms are very different. Hebbian learning is unsupervised. LMS learning is supervised. However, a form of LMS can be constructed to perform unsupervised learning and, as such, LMS can be used in a natural way to implement Hebbian learning. Combining the two paradigms creates a new unsupervised learning algorithm, Hebbian-LMS. This algorithm has practical engineering applications and provides insight into learning in living neural networks. A fundamental question is how does learning take place in living neural networks? “Nature's little secret,” the learning algorithm practiced by nature at the neuron and synapse level, may well be the Hebbian-LMS algorithm.

Keywords

Adaptive filtering; Bootstrap learning; Clustering; Decision-directed learning; Hebbian learning; Hebbian-LMS algorithm; LMS algorithm; Neural networks; Synaptic plasticity

Chapter Outline

1. 1. Introduction
2. 2. ADALINE and the LMS Algorithm, From the 1950s
3. 3. Unsupervised Learning With Adaline, From the 1960s
4. 4. Robert Lucky's Adaptive Equalization, From the 1960s
5. 5. Bootstrap Learning With a Sigmoidal Neuron
6. 6. Bootstrap Learning With a More “Biologically Correct” Sigmoidal Neuron
   1. 6.1 Training a Network of Hebbian-LMS Neurons
7. 7. Other Clustering Algorithms
   1. 7.1 K-Means Clustering
   2. 7.2 Expectation-Maximization Algorithm
   3. 7.3 Density-Based Spatial Clustering of Application With Noise Algorithm
   4. 7.4 Comparison Between Clustering Algorithms
8. 8. A General Hebbian-LMS Algorithm
9. 9. The Synapse
10. 10. Postulates of Synaptic Plasticity
11. 11. The Postulates and the Hebbian-LMS Algorithm
12. 12. Nature's Hebbian-LMS Algorithm
13. 13. Conclusion
14. Appendix: Trainable Neural Network Incorporating Hebbian-LMS Learning
15. Acknowledgments
16. References

1. Introduction

Donald O. Hebb has had considerable influence in the fields of psychology and neurobiology since the publication of his book The Organization of Behavior in 1949 [1]. Hebbian learning is often described as: “neurons that fire together wire together.” Now imagine a large network of interconnected neurons whose synaptic weights are increased because the presynaptic neuron and the postsynaptic neuron fired together. This might seem strange. What purpose would nature fulfill with such a learning algorithm?

In his book, Hebb actually said: “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.”

“Fire together wire together” is a simplification of this. Wire together means increase the synaptic weight. Fire together is not exactly what Hebb said, but some researchers have taken this literally and believe that information is carried with the timing of each activation pulse. Some believe that the precise timing of presynaptic and postsynaptic firings has an effect on synaptic weight changes. There is some evidence for these ideas [2–4] but they remain controversial.

Neuron-to-neuron signaling in the brain is done with pulse trains. This is AC coupling and is one of nature's “good ideas,” avoiding the effects of DC level drift that could be caused by the presence of fluids and electrolytes in the brain. We believe that the output signal of a neuron is the neuron's firing rate as a function of time.

Neuron-to-neuron signaling in computer simulated artificial neural networks is done in most cases with DC levels. If a static input pattern vector is presented, the neuron's output is an analog DC level that remains constant as long as the input pattern vector is applied. That analog output can be weighted by a synapse and applied as an input to another neuron, a “postsynaptic” neuron, in a layered network or otherwise interconnected network.

The purpose of this chapter is to review a new learning algorithm that we call Hebbian-LMS [5]. It is an implementation of Hebb's teaching by means of the LMS algorithm of Widrow and Hoff. With the Hebbian LMS algorithm, unsupervised or autonomous learning takes place locally, in the individual neuron and its synapses, and when many such neurons are connected in a network, the entire network learns autonomously. One might ask, “what does it learn?” This question will be considered below where applications will be presented.

There is another question that can be asked: “Should we believe in Hebbian learning? Did Hebb arrive at this idea by doing definitive biological experiments, by ‘getting his hands wet’”? The answer is no. The idea came to him by intuitive reasoning. Like Newton's theory of gravity, like Einstein's theory of relativity, like Darwin's theory of evolution, it was a thought experiment propounded long before modern knowledge and instrumentation could challenge it, refute it, o...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• List of Contributors
• Editors' Brief Biographies
• Introduction
• Chapter 1. Nature's Learning Rule: The Hebbian-LMS Algorithm
• Chapter 2. A Half Century of Progress Toward a Unified Neural Theory of Mind and Brain With Applications to Autonomous Adaptive Agents and Mental Disorders
• Chapter 3. Third Gen AI as Human Experience Based Expert Systems
• Chapter 4. The Brain-Mind-Computer Trichotomy: Hermeneutic Approach
• Chapter 5. From Synapses to Ephapsis: Embodied Cognition and Wearable Personal Assistants
• Chapter 6. Evolving and Spiking Connectionist Systems for Brain-Inspired Artificial Intelligence
• Chapter 7. Pitfalls and Opportunities in the Development and Evaluation of Artificial Intelligence Systems
• Chapter 8. The New AI: Basic Concepts, and Urgent Risks and Opportunities in the Internet of Things
• Chapter 9. Theory of the Brain and Mind: Visions and History
• Chapter 10. Computers Versus Brains: Game Is Over or More to Come?
• Chapter 11. Deep Learning Approaches to Electrophysiological Multivariate Time-Series Analysis
• Chapter 12. Computational Intelligence in the Time of Cyber-Physical Systems and the Internet of Things
• Chapter 13. Multiview Learning in Biomedical Applications
• Chapter 14. Meaning Versus Information, Prediction Versus Memory, and Question Versus Answer
• Chapter 15. Evolving Deep Neural Networks
• Index