Combining Neural Networks With Unsupervised Learning
Topic 6e introduced neural networks as supervised learning systems: trained on labeled examples, learning to map inputs to correct outputs. Topic 7b introduced unsupervised learning as learning without labels: finding structure in data through clustering and dimensionality reduction.
These two ideas can be combined. A neural network can be trained without labeled data, using a training signal derived from the data itself. The result is a class of models that learns rich internal representations of data without any human annotation — and those representations turn out to be remarkably useful.
Autoencoders: Learning by Reconstruction
An autoencoder is a neural network trained to reproduce its own input as its output. That sounds trivial — surely any network can just copy the input to the output? The trick is the constraint: between the input and output layers, the network must pass through a bottleneck — a hidden layer with far fewer neurons than the input or output.
The autoencoder compresses input through a narrow bottleneck, then reconstructs it. To reconstruct accurately, the encoding must capture what matters most.
The training task is: make the output as similar to the input as possible. The error signal is the difference between input and output — no labels needed. Because the network must pass through the bottleneck, it is forced to find an efficient compressed representation: it cannot just copy the input through, so it must learn which features are most important for reconstruction.
That compressed representation — the encoding — is often the most valuable output of the autoencoder. It is a compact summary of the input that preserves its essential structure, discovered without any human labeling. Once the autoencoder is trained, the encoding layer can be used as a reduced-dimension representation of the data, fed into clustering algorithms, visualized with t-SNE, or used to initialize a supervised classifier.
Unsupervised Pretraining: Learning From Everything
The most consequential application of unsupervised learning in modern AI is pretraining: training a large neural network on a massive corpus of unlabeled data before fine-tuning it on a specific supervised task.
The logic is compelling. Labeled data is scarce and expensive to produce. Unlabeled data is essentially unlimited — the entire internet, all digitized books, all text ever recorded. If a network can learn rich, general representations of the world from unlabeled data, those representations can then be adapted for specific tasks with relatively little labeled data.
The earliest successful pretraining approaches were autoencoders and similar reconstruction-based models. Researchers discovered that a network pretrained to reconstruct its inputs had learned internal representations that generalized remarkably well to supervised tasks it had never seen during pretraining. The bottleneck forced it to learn the structure of the data, not just to memorize specific examples.
The Next-Word Prediction Insight
For text data, the most powerful unsupervised pretraining task turned out to be surprisingly simple: given a sequence of words, predict the next word.
This is not exactly a supervised task (no human labeled the "correct" next word) and not exactly unsupervised (the label is derived from the data itself — the actual next word in the text). It is sometimes called self-supervised learning. The training signal is abundant because every sentence in every document provides thousands of prediction opportunities. A corpus of 100 billion words provides 100 billion training examples, all generated automatically.
What makes next-word prediction so powerful is that doing it well requires understanding language at every level: grammar, word meaning, sentence structure, topic coherence, and real-world knowledge. A network that has learned to predict the next word accurately across billions of diverse texts has necessarily learned an enormous amount about how language and the world work — not because anyone taught it these things explicitly, but because they are required to predict text well.
The next-word prediction task is, on its surface, trivial. Its consequence is a network that has internalized a vast, compressed model of language and the world — without a single human label. That is the foundation on which modern large language models are built.
The Bridge to Topic 7c
Topic 7c covers large language models — ChatGPT, Claude, and their relatives. These systems are the most visible AI technology in the world today, and they are the subject of enormous public debate, much of it based on significant misunderstanding of how they work.
You are now equipped with the foundational concepts to understand them properly:
- They are neural networks (Topic 6e) — trained by adjusting billions of weights to minimize prediction error.
- They are pretrained using unsupervised self-supervised learning (this reading) — next-word prediction on massive text corpora, with no human labels.
- They are then fine-tuned using supervised learning and reinforcement learning from human feedback (Topics 6c and 7b) — shaped by human ratings of their outputs to make them more helpful, harmless, and honest.
- They find patterns in data that are invisible to direct inspection (Topic 7b) — representing language in a high-dimensional space that can be probed and visualized but not directly read.
None of that makes LLMs simple. But it makes them comprehensible. Topic 7c is where that comprehension becomes consequential for your classroom.