3. Neural Networks

1943 Foundations of Neural Networks

Laid the theoretical groundwork for ANNs, inspiring future computational models of the brain.

Warren McCulloch and Walter Pitts publish a paper introducing the first mathematical model of a neural network, describing neurons as logical decision-making units.

McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity.

Ch. 4 · Perceptron

1950 Turing Test Proposed

Established a benchmark for assessing AI capabilities, influencing the philosophical and practical development of AI.

Alan Turing publishes "Computing Machinery and Intelligence," proposing the Turing Test to evaluate a machine's ability to exhibit intelligent behavior indistinguishable from that of a human.

Turing, A. M. (1950). Computing Machinery and Intelligence.

1952 Hodgkin–Huxley Model

Foundational for understanding neural dynamics, it influenced the development of artificial neural networks.

Alan Hodgkin and Andrew Huxley develop a mathematical model of the action potential in neurons, describing how neurons transmit signals through electrical impulses.

Hodgkin, A. L., & Huxley, A. F. (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve.

1956 Birth of AI as a Discipline

Marked the formal establishment of AI as a field of study, fostering research into machine intelligence.

The Dartmouth Conference, organized by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon, coins the term artificial intelligence.

McCarthy, J., Minsky, M., Rochester, N., & Shannon, C. (1955). Dartmouth Conference Proposal.

1958 Perceptron Introduced

Pioneered the concept of a simple neural network, laying the foundation for future developments in machine learning and neural networks.

Frank Rosenblatt develops the Perceptron, an early artificial neural network capable of learning to classify patterns. It consists of a single layer of output nodes connected to input features, using a step function to produce binary outputs.

Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.

Ch. 4 · Perceptron

1964 Visual Cortex and Monocular Deprivation

Highlighted the importance of experience in neural plasticity and the hierarchical organisation of the visual cortex.

David H. Hubel and Torsten N. Wiesel conduct pioneering research on the visual cortex of cats. Their work on monocular deprivation shows that depriving one eye of visual input during a critical period leads to permanent changes in the visual cortex.

Hubel, D. H., & Wiesel, T. N. (1964). Effects of monocular deprivation in kittens.

Ch. 10 · Convolutional Networks

1969 Limitations of Perceptrons → AI Winter

Highlighted the limitations of early neural networks, leading to a temporary decline in interest in neural networks and AI.

Marvin Minsky and Seymour Papert publish "Perceptrons," critiquing the limitations of single-layer perceptrons, particularly their inability to solve non-linearly separable problems like the XOR problem. The book contributed to a decline in neural network research that lasted over a decade, the first "AI winter" for the field, as funding and interest shifted to symbolic AI.

Minsky, M., & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry.

Ch. 5 · Multi-Layer Perceptron

1980 Neocognitron

A precursor to modern CNNs, it demonstrated the potential of hierarchical feature extraction in image recognition.

Kunihiko Fukushima develops the Neocognitron, an early convolutional neural network model that mimics the hierarchical structure of the visual cortex.

Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position.

Ch. 10 · Convolutional Networks

1982 Hopfield Networks

Laid the groundwork for later developments in neural network architectures and energy-based models.

John Hopfield introduces Hopfield networks, a type of recurrent neural network that serves as an associative memory, capable of storing and recalling patterns.

Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities.

1982 Self-Organizing Maps (SOM)

Provided an unsupervised way to cluster and visualise complex, high-dimensional data.

Teuvo Kohonen develops Self-Organizing Maps, an unsupervised learning algorithm that maps high-dimensional data onto a lower-dimensional grid.

Kohonen, T. (1982). Self-organized formation of topologically correct feature maps.

1986 Backpropagation Rediscovered

Revived interest in ANNs by overcoming limitations of single-layer perceptrons, paving the way for deep learning.

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams publish a paper on backpropagation, enabling training of multi-layer neural networks.

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors.

Ch. 5 · Multi-Layer Perceptron

1989 Universal Approximation Theorem

Established the theoretical foundation for neural networks' ability to approximate any continuous function, leading to the development of deep learning.

George Cybenko proves that a feedforward neural network with a single hidden layer can approximate any continuous function on compact subsets of \(\mathbb{R}^n\) under mild conditions on the activation function.

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function.

Ch. 5 · Multi-Layer Perceptron

1997 Deep Blue Defeats Chess Champion

Showcased the potential of AI in strategic games, leading to advancements in game-playing AI.

IBM's Deep Blue defeats world chess champion Garry Kasparov in a six-game match, marking a significant milestone in AI's ability to compete with human intelligence in complex tasks.

Campbell, M., Hoane, A. J., & Hsu, F. (2002). Deep Blue.

1997 Long Short-Term Memory (LSTM)

Addressed the vanishing gradient problem in RNNs, enabling effective modeling of long-term dependencies in sequential data.

Sepp Hochreiter and Jürgen Schmidhuber introduce LSTM networks, a type of recurrent neural network designed to learn long-term dependencies in sequential data.

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory.

Ch. 11 · Attention

1998 Convolutional Neural Networks (CNNs)

Revolutionized computer vision and image processing, enabling breakthroughs in object recognition and classification.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner publish a paper on CNNs trained by gradient descent, presenting the LeNet-5 architecture for handwritten digit recognition.

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition.

Ch. 10 · Convolutional Networks

2006 Deep Learning Renaissance

Sparked the modern deep learning era by showing that deep networks could be trained efficiently.

Geoffrey Hinton and colleagues introduce deep belief networks, demonstrating effective pre-training for deep neural networks.

Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets.

Ch. 16 · Generative Models

2012 AlexNet and the ImageNet Breakthrough

Demonstrated the superiority of deep learning in computer vision, leading to widespread adoption.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton’s AlexNet wins the ImageNet competition, achieving unprecedented accuracy in image classification using deep CNNs.

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks.

Ch. 10 · Convolutional Networks

2013 word2vec and Word Embeddings

Showed that a shallow network trained to predict neighbouring words learns vectors whose geometry encodes meaning.

Tomas Mikolov and colleagues at Google train two simple models (CBOW and skip-gram) on billions of words. The resulting embeddings answer analogies such as king − man + woman ≈ queen and become the standard input layer of NLP models.

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space.

Definitions · Embedding

2013 Variational Autoencoders (VAEs)

Turned the autoencoder into a probabilistic generative model that can be trained by gradient descent.

Diederik Kingma and Max Welling introduce the VAE and the reparameterization trick, which lets the gradient pass through a random sample: an encoder maps each input to a distribution over a latent space and a decoder generates data from points sampled in it.

Kingma, D. P., & Welling, M. (2013). Auto-encoding variational Bayes.

Ch. 17 · Variational Autoencoders

2014 Generative Adversarial Networks (GANs)

Introduced a novel approach to generative modeling, enabling the creation of realistic synthetic data.

Ian Goodfellow and colleagues introduce GANs, a framework for training generative models using adversarial training.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets.

Ch. 18 · GANs

2014 Attention for Neural Machine Translation

Let a network look back at every input position instead of squeezing a whole sentence into one vector: the idea the Transformer is built on.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio add an attention mechanism to a recurrent encoder–decoder: for each output word, the decoder computes weights over all encoder states and reads their weighted average.

Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate.

Ch. 11 · Attention

2015 Residual Networks (ResNets)

Enabled the training of very deep neural networks, significantly improving performance on image recognition tasks.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun introduce Residual Networks, which use skip connections to let gradients flow easily through deep networks.

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition.

Ch. 9 · Deep Networks

2016 DeepMind’s AlphaGo

Showcased deep learning’s ability to tackle complex strategic games, advancing AI research.

DeepMind’s AlphaGo, using deep reinforcement learning and neural networks, defeats professional Go player Lee Sedol.

Silver, D., et al. (2016). Mastering the game of Go with deep neural networks and tree search.

2017 Transformers and Attention Mechanisms

Revolutionized natural language processing and sequence modeling, enabling breakthroughs in machine translation and text generation.

Ashish Vaswani and colleagues introduce the Transformer architecture, which uses self-attention mechanisms to process sequences in parallel, significantly improving performance in NLP tasks.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need.

Ch. 12 · Transformers

2018 BERT and Pre-trained Language Models

Set new standards in NLP by introducing pre-training and fine-tuning techniques, enabling models to understand context and semantics better.

Jacob Devlin and colleagues introduce BERT (Bidirectional Encoder Representations from Transformers), a pre-trained language model that achieves state-of-the-art results on various NLP benchmarks.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding.

Ch. 14 · Transfer Learning

2020 GPT-3 and Large Language Models

Showcased the capabilities of large-scale language models, enabling advancements in natural language understanding and generation.

OpenAI releases GPT-3, a 175 billion parameter language model, demonstrating impressive performance in various NLP tasks, including text generation, translation, and question answering.

Brown, T. B., Mann, B., Ryder, N., Subbiah, S., Kaplan, J., Dhariwal, P., Neelakantan, S., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, D., Litwin, M., Gray, S., Chess, B., Clark, J., Berridge, S., Zaremba, W., & Amodei, D. (2020). Language models are few-shot learners.

Ch. 15 · Large Language Models

2020 Denoising Diffusion Models

Made diffusion a practical generative model, soon the basis of the leading image generators.

Jonathan Ho, Ajay Jain, and Pieter Abbeel train a network to remove a little Gaussian noise at a time; generating an image means starting from pure noise and denoising it step by step.

Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models.

Ch. 20 · Stable Diffusion

2020 Vision Transformer (ViT)

Showed that a plain Transformer, with no convolutions, matches the best CNNs when trained on enough data.

Alexey Dosovitskiy and colleagues cut an image into 16×16 patches, treat each patch as a token and feed the sequence to a standard Transformer encoder.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale.

Ch. 13 · Vision Transformers

2021 DALL-E and Image Generation

Enabled the generation of high-quality images from textual descriptions, showcasing the potential of multimodal AI.

OpenAI introduces DALL-E, a model capable of generating images from textual descriptions, demonstrating the power of combining language and vision.

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021). Zero-Shot Text-to-Image Generation.

Ch. 23 · Autoregressive Generation

2021 CLIP: Images and Text in One Space

Learned one embedding space for images and text from 400 million captioned images, enabling zero-shot classification and guiding text-to-image models.

Alec Radford and colleagues at OpenAI train an image encoder and a text encoder together with a contrastive loss: matching image–caption pairs are pulled together and mismatched ones pushed apart.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision.

Ch. 19 · CLIP

2022 Latent Diffusion and Stable Diffusion

Moved diffusion into the compressed latent space of an autoencoder, making high-resolution text-to-image generation cheap enough to run on a single GPU.

Robin Rombach and colleagues run the diffusion process on the latent codes of a pretrained autoencoder and condition it on text through cross-attention. Stable Diffusion, released openly in August 2022, is built on this model.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models.

Ch. 20 · Stable Diffusion

2022 InstructGPT, RLHF and ChatGPT

Turned a language model into an assistant that follows instructions, taking LLMs to the general public.

Long Ouyang and colleagues fine-tune GPT-3 on human demonstrations and then with reinforcement learning from human feedback (RLHF), using a reward model trained on people's rankings of answers. ChatGPT, released in November 2022, follows the same recipe.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback.

Ch. 15 · Large Language Models

2024 Nobel Prizes for Neural Networks

Recognised artificial neural networks as a contribution to science beyond computing.

The Nobel Prize in Physics goes to John Hopfield and Geoffrey Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks”. Half of the Chemistry prize goes to Demis Hassabis and John Jumper for AlphaFold, a deep network that predicts the 3D structure of a protein from its amino-acid sequence (the other half goes to David Baker, for computational protein design).

Jumper, J., et al. (2021). Highly accurate protein structure prediction with AlphaFold.