12. Generative Models
Generative AI, often abbreviated as GenAI, is a subfield of artificial intelligence that employs generative models to create new content, such as text, images, videos, audio, software code, or other data forms, by learning patterns and structures from vast training datasets. These models typically respond to user prompts (e.g., natural language inputs) by producing original outputs that mimic the style or characteristics of the learned data, distinguishing them from traditional AI systems that primarily analyze or predict existing information. Common examples include tools like ChatGPT for text generation and DALL-E for image creation, powered by techniques such as large language models (LLMs) or generative adversarial networks (GANs).
Generative AI's roots trace back to early probability models, evolving through rule-based systems to deep learning. We'll cover chronologically, with detailed highlights on transformative events.
| Era/Year | Key Development | Details/Highlights | Impact |
|---|---|---|---|
| 1950s: Probabilistic Foundations | Markov Chains (1953, Claude Shannon) | Shannon's work on information theory introduced Markov models for text generation (e.g., predicting next letters). Highlight: First "AI poem" generated via chains—crude but proved machines could mimic patterns. | Laid groundwork for sequence generation; influenced NLP. |
| 1980s: Early Neural Nets | Boltzmann Machines (1986, Geoffrey Hinton et al.) | Restricted Boltzmann Machines (RBMs) used energy-based models to learn data distributions. Highlight: Hinton's "wake-sleep" algorithm (1995 precursor) trained unsupervised nets on images—first glimpses of generative "dreaming." | Bridge to deep learning; used in early recommender systems (e.g., Netflix Prize roots). |
| 1990s: Variational Inference | Variational Autoencoders (VAEs) precursors (1990s, Dayan et al.) | Bayesian methods for latent variable models. Highlight: Jordan & Weiss's variational EM (1998)—key paper enabling tractable posterior approximations. | Enabled scalable generative modeling; foundation for modern VAEs. |
| 2010s: Deep Learning Boom | Deep Belief Nets (2006, Hinton) → VAEs (2013, Kingma & Welling) | Stacked RBMs pre-trained deep nets. Highlight: VAEs paper (ICLR 2014) introduced reparameterization trick for backprop through stochastic nodes—generated blurry MNIST digits, but scalable. Diffusion models' roots in score-based generative modeling (Sohl-Dickstein et al., 2015). | Democratized unsupervised learning; VAEs in drug discovery (e.g., AlphaFold precursors). |
| 2014-Present: Adversarial Era | GANs (2014, Goodfellow et al.) | Subsequent: StyleGAN (2018, NVIDIA) for photorealistic faces. Highlight: GANs' NIPS 2014 debut generated realistic bedrooms—shocked community, sparking "adversarial training" paradigm. AlphaGo (2016) used generative rollouts. GPT-1 (2018) for text; | Exploded applications (e.g., DeepFakes 2017); ethical debates (e.g., 2018 EU AI ethics guidelines). |
| 2020s: Scaling & Multimodal | Diffusion Models (2020, Ho et al.); GPT-3 (2020, OpenAI) | Denoising diffusion probabilistic models (DDPMs). Highlight: DALL·E (2021) combined diffusion + transformers for text-to-image; Stable Diffusion (2022) open-sourced, generating 1B+ images/month. | Multimodal gen AI (e.g., Sora video gen, 2024); concerns over energy use (training GPT-4 ~1GWh). |
-
Geeks for Geeks - What is Generative AI? ↩