Transformers for Machine Learning
eBook - ePub

Transformers for Machine Learning

A Deep Dive

Uday Kamath, Kenneth Graham, Wael Emara

  1. 257 pages
  2. English
  3. ePUB (adapté aux mobiles)
  4. Disponible sur iOS et Android
eBook - ePub

Transformers for Machine Learning

A Deep Dive

Uday Kamath, Kenneth Graham, Wael Emara

DĂ©tails du livre
Aperçu du livre
Table des matiĂšres

À propos de ce livre

Transformers are becoming a core part of many neural network architectures, employed in a wide range of applications such as NLP, Speech Recognition, Time Series, and Computer Vision. Transformers have gone through many adaptations and alterations, resulting in newer techniques and methods. Transformers for Machine Learning: A Deep Dive is the first comprehensive book on transformers.

Key Features:

  • A comprehensive reference book for detailed explanations for every algorithm and techniques related to the transformers.
  • 60+ transformer architectures covered in a comprehensive manner.
  • A book for understanding how to apply the transformer techniques in speech, text, time series, and computer vision.
  • Practical tips and tricks for each architecture and how to use it in the real world.
  • Hands-on case studies and code snippets for theory and practical real-world analysis using the tools and libraries, all ready to run in Google Colab.

The theoretical explanations of the state-of-the-art transformer architectures will appeal to postgraduate students and researchers (academic and industry) as it will provide a single entry point with deep discussions of a quickly moving field. The practical hands-on case studies and code will appeal to undergraduate students, practitioners, and professionals as it allows for quick experimentation and lowers the barrier to entry into the field.

Foire aux questions

Comment puis-je résilier mon abonnement ?
Il vous suffit de vous rendre dans la section compte dans paramĂštres et de cliquer sur « RĂ©silier l’abonnement ». C’est aussi simple que cela ! Une fois que vous aurez rĂ©siliĂ© votre abonnement, il restera actif pour le reste de la pĂ©riode pour laquelle vous avez payĂ©. DĂ©couvrez-en plus ici.
Puis-je / comment puis-je télécharger des livres ?
Pour le moment, tous nos livres en format ePub adaptĂ©s aux mobiles peuvent ĂȘtre tĂ©lĂ©chargĂ©s via l’application. La plupart de nos PDF sont Ă©galement disponibles en tĂ©lĂ©chargement et les autres seront tĂ©lĂ©chargeables trĂšs prochainement. DĂ©couvrez-en plus ici.
Quelle est la différence entre les formules tarifaires ?
Les deux abonnements vous donnent un accĂšs complet Ă  la bibliothĂšque et Ă  toutes les fonctionnalitĂ©s de Perlego. Les seules diffĂ©rences sont les tarifs ainsi que la pĂ©riode d’abonnement : avec l’abonnement annuel, vous Ă©conomiserez environ 30 % par rapport Ă  12 mois d’abonnement mensuel.
Qu’est-ce que Perlego ?
Nous sommes un service d’abonnement Ă  des ouvrages universitaires en ligne, oĂč vous pouvez accĂ©der Ă  toute une bibliothĂšque pour un prix infĂ©rieur Ă  celui d’un seul livre par mois. Avec plus d’un million de livres sur plus de 1 000 sujets, nous avons ce qu’il vous faut ! DĂ©couvrez-en plus ici.
Prenez-vous en charge la synthÚse vocale ?
Recherchez le symbole Écouter sur votre prochain livre pour voir si vous pouvez l’écouter. L’outil Écouter lit le texte Ă  haute voix pour vous, en surlignant le passage qui est en cours de lecture. Vous pouvez le mettre sur pause, l’accĂ©lĂ©rer ou le ralentir. DĂ©couvrez-en plus ici.
Est-ce que Transformers for Machine Learning est un PDF/ePUB en ligne ?
Oui, vous pouvez accĂ©der Ă  Transformers for Machine Learning par Uday Kamath, Kenneth Graham, Wael Emara en format PDF et/ou ePUB ainsi qu’à d’autres livres populaires dans Informatica et Reti neurali. Nous disposons de plus d’un million d’ouvrages Ă  dĂ©couvrir dans notre catalogue.


Reti neurali

CHAPTER 1Deep Learning and Transformers: An Introduction

DOI: 10.1201/9781003170082-1
TRANSFORMERS are deep learning models that have achieved state-of-the-art performance in several fields such as natural language processing, computer vision, and speech recognition. Indeed, the massive surge of recently proposed transformer model variants has meant researchers and practitioners alike find it challenging to keep pace. In this chapter, we provide a brief history of diverse research directly or indirectly connected to the innovation of transformers. Next, we discuss a taxonomy based on changes in the architecture for efficiency in computation, memory, applications, etc., which can help navigate the complex innovation space. Finally, we provide resources in tools, libraries, books, and online courses that the readers can benefit from in their pursuit.

1.1 Deep Learning: A Historic Perspective

In the early 1940s, S. McCulloch and W. Pitts, using a simple electrical circuit called a “threshold logic unit”, simulated intelligent behavior by emulating how the brain works [179]. The simple model had the first neuron with inputs and outputs that would generate an output 0 when the “weighted sum” was below a threshold and 1 otherwise, which later became the basis of all the neural architectures. The weights were not learned but adjusted. In his book The Organization of Behaviour (1949), Donald Hebb laid the foundation of complex neural processing by proposing how neural pathways can have multiple neurons firing and strengthening over time [108]. Frank Rosenblatt, in his seminal work, extended the McCulloch–Pitts neuron, referring to it as the “Mark I Perceptron”; given the inputs, it generated outputs using linear thresholding logic [212].
The weights in the perceptron were “learned” by repeatedly passing the inputs and reducing the difference between the predicted output and the desired output, thus giving birth to the basic neural learning algorithm. Marvin Minsky and Seymour Papert later published the book Perceptrons which revealed the limitations of perceptrons in learning the simple exclusive-or function (XOR) and thus prompting the so-called The First AI Winter [186].
John Hopfield introduced “Hopfield Networks”, one of the first recurrent neural networks (RNNs) that serve as a content-addressable memory system [117].
In 1986, David Rumelhart, Geoff Hinton, and Ronald Williams published the seminal work “Learning representations by back-propagating errors” [217]. Their work confirms how a multi-layered neural network using many “hidden” layers can overcome the weakness of perceptrons in learning complex patterns with relatively simple training procedures. The building blocks for this work had been laid down by various research over the years by S. Linnainmaa, P. Werbos, K. Fukushima, D. Parker, and Y. LeCun [91, 149, 164, 196, 267].
LeCun et al., through their research and implementation, led to the first widespread application of neural networks to recognize the hand-written digits used by the U.S. Postal Service [150]. This work is a critical milestone in deep learning history, proving the utility of convolution operations and weight sharing in learning the features in computer vision.
Backpropagation, the key optimization technique, encountered a number of issues such as vanishing gradients, exploding gradients, and the inability to learn long-term information, to name a few [115]. Hochreiter and Schmidhuber, in their work,“Long short-term memory (LSTM)” architecture, demonstrated how issues with long-term dependencies could overcome shortcomings of backpropagation over time [116].
Hinton et al. published a breakthrough paper in 2006 titled “A fast learning algorithm for deep belief nets”; it was one of the reasons for the resurgence of deep learning [113]. The research highlighted the effectiveness of layer-by-layer training using unsupervised methods followed by supervised “fine-tuning” to achieve state-of-the-art results in character recognition. Bengio et al., in their seminal work following this, offered deep insights into why deep learning networks with multiple layers can hierarchically learn features as compared to shallow neural networks [27]. In their research, Bengio and LeCun emphasized the advantages of deep learning through architectures such as convolutional neural networks (CNNs), restricted Boltzmann machines (RBMs), and deep belief networks (DBNs), and through techniques such as unsupervised pre-training with fine-tuning, thus inspiring the next wave of deep learning [28]. Fei-Fei Li, head of the artificial intelligence lab at Stanford University, along with other researchers, launched ImageNet, which resulted in the most extensive collection of images and, for the first time, highlighted the usefulness of data in learning essential tasks such as object ...

Table des matiĂšres

  1. Cover Page
  2. Half-Title Page
  3. Series Page
  4. Title Page
  5. Copyright Page
  6. Dedication Page
  7. Contents
  8. Foreword
  9. Preface
  10. Authors
  11. Contributors
  12. Chapter 1 Deep Learning and Transformers: An Introduction
  13. Chapter 2 Transformers: Basics and Introduction
  14. Chapter 3 Bidirectional Encoder Representations from Transformers (BERT)
  15. Chapter 4 Multilingual Transformer Architectures
  16. Chapter 5 Transformer Modifications
  17. Chapter 6 Pre-trained and Application-Specific Transformers
  18. Chapter 7 Interpretability and Explainability Techniques for Transformers
  19. Bibliography
  20. Index
Normes de citation pour Transformers for Machine Learning

APA 6 Citation

Kamath, U., Graham, K., & Emara, W. (2022). Transformers for Machine Learning (1st ed.). CRC Press. Retrieved from (Original work published 2022)

Chicago Citation

Kamath, Uday, Kenneth Graham, and Wael Emara. (2022) 2022. Transformers for Machine Learning. 1st ed. CRC Press.

Harvard Citation

Kamath, U., Graham, K. and Emara, W. (2022) Transformers for Machine Learning. 1st edn. CRC Press. Available at: (Accessed: 15 October 2022).

MLA 7 Citation

Kamath, Uday, Kenneth Graham, and Wael Emara. Transformers for Machine Learning. 1st ed. CRC Press, 2022. Web. 15 Oct. 2022.