Introduction

Foundations of Neural Networks

Laid the theoretical groundwork for ANNs, inspiring future computational models of the brain.
Warren McCulloch and Walter Pitts publish a paper introducing the first mathematical model of a neural network, describing neurons as logical decision-making units.

McCulloch, & W. S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity.

Turing Test Proposed

Established a benchmark for assessing AI capabilities, influencing the philosophical and practical development of AI.
Alan Turing publishes "Computing Machinery and Intelligence," proposing the Turing Test to evaluate a machine's ability to exhibit intelligent behavior indistinguishable from that of a human.

Turing, A. M. (1950). Computing Machinery and Intelligence.

Birth of AI as a Discipline

Marked the formal establishment of AI as a field of study, fostering research into machine intelligence.
The Dartmouth Conference, organized by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon, coins the term artificial intelligence.

McCarthy, J., Minsky, M., Rochester, N., Shannon, C. (1955). Dartmouth Conference Proposal.

Perceptron Introduced

Pioneered the concept of a simple neural network, laying the foundation for future developments in machine learning and neural networks.
Frank Rosenblatt develops the Perceptron, an early artificial neural network capable of learning to classify patterns. It consists of a single layer of output nodes connected to input features, using a step function to produce binary outputs.

Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.

Limitations of Perceptrons → AI Winter

Highlighted the limitations of early neural networks, leading to a temporary decline in interest in neural networks and AI.
Marvin Minsky and Seymour Papert publish "Perceptrons," critiquing the limitations of single-layer perceptrons, particularly their inability to solve non-linearly separable problems like the XOR problem. This work leads to a decline in neural network research for over a decade. Led to the first "AI winter," a period of reduced funding and interest in neural networks, shifting focus to symbolic AI.

Minsky, M., Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry.

Backpropagation Rediscovered

Revived interest in ANNs by overcoming limitations of single-layer perceptrons, paving the way for deep learning.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams publish a paper on backpropagation, enabling training of multi-layer neural networks.

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors.

Universal Approximation Theorem

Established the theoretical foundation for neural networks' ability to approximate any continuous function, leading to the development of deep learning.
George Cybenko proves that a feedforward neural network with a single hidden layer can approximate any continuous function on compact subsets of \(R^n\) under mild conditions on the activation function.

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function.

Deep Blue Defeats Chess Champion

Showcased the potential of AI in strategic games, leading to advancements in game-playing AI and deep learning.
IBM's Deep Blue defeats world chess champion Garry Kasparov in a six-game match, marking a significant milestone in AI's ability to compete with human intelligence in complex tasks.

Campbell, M., Hoane, A. J., & Hsu, F. (2002). Deep Blue.

Convolutional Neural Networks (CNNs)

Revolutionized computer vision and image processing, enabling breakthroughs in object recognition and classification.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton publish a paper on CNNs, introducing the LeNet architecture for handwritten digit recognition.

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition.

Deep Learning Renaissance

Sparked the modern deep learning era by showing that deep networks could be trained efficiently.
Geoffrey Hinton and colleagues introduce deep belief networks, demonstrating effective pre-training for deep neural networks.

Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets.

AlexNet and the ImageNet Breakthrough

Demonstrated the superiority of deep learning in computer vision, leading to widespread adoption.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton’s AlexNet wins the ImageNet competition, achieving unprecedented accuracy in image classification using deep CNNs.

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks.

Generative Adversarial Networks (GANs)

Introduced a novel approach to generative modeling, enabling the creation of realistic synthetic data.
Ian Goodfellow and colleagues introduce GANs, a framework for training generative models using adversarial training.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets.

DeepMind’s AlphaGo

Showcased deep learning’s ability to tackle complex strategic games, advancing AI research.
DeepMind’s AlphaGo, using deep reinforcement learning and neural networks, defeats professional Go player Lee Sedol.

Silver, D., et al. (2016). Mastering the game of Go with deep neural networks and tree search.

Transformers and Attention Mechanisms

Revolutionized natural language processing and sequence modeling, enabling breakthroughs in machine translation and text generation.
Ashish Vaswani and colleagues introduce the Transformer architecture, which uses self-attention mechanisms to process sequences in parallel, significantly improving performance in NLP tasks.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need.

BERT and Pre-trained Language Models

Set new standards in NLP by introducing pre-training and fine-tuning techniques, enabling models to understand context and semantics better.
Jacob Devlin and colleagues introduce BERT (Bidirectional Encoder Representations from Transformers), a pre-trained language model that achieves state-of-the-art results on various NLP benchmarks.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding.

GPT-3 and Large Language Models

Showcased the capabilities of large-scale language models, enabling advancements in natural language understanding and generation.
OpenAI releases GPT-3, a 175 billion parameter language model, demonstrating impressive performance in various NLP tasks, including text generation, translation, and question answering.

Brown, T. B., Mann, B., Ryder, N., Subbiah, S., Kaplan, J., Dhariwal, P., Neelakantan, S., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, D., Litwin, M., Gray, S., Chess, B., Clark, J., Berridge, S., Zaremba, W., & Amodei, D. (2020). Language models are few-shot learners.

DALL-E and Image Generation

Enabled the generation of high-quality images from textual descriptions, showcasing the potential of multimodal AI.
OpenAI introduces DALL-E, a model capable of generating images from textual descriptions, demonstrating the power of combining language and vision.

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., & Sutskever, I. (2021). Zero-Shot Text-to-Image Generation.