Topic 6e, Reading 2: Networks and Learning

Building a Network From Layers

A single perceptron produces a binary output based on a weighted combination of its inputs. Connect many perceptrons together and you get a network capable of representing far more complex relationships between inputs and outputs.

Artificial neural networks are organized into layers. Every network has at least two: an input layer and an output layer. Most useful networks also have one or more hidden layers in between.

The input layer receives the raw data. Each neuron in this layer corresponds to one feature of the input. For a spam filter, the input layer might have one neuron for each possible signal: presence of certain words, length of the message, whether the sender is known, etc. For an image classifier, each input neuron corresponds to one pixel.
Hidden layers sit between the input and output layers. Their neurons receive the outputs of the previous layer, process them, and pass results forward. Hidden layers are where the network builds internal representations of the data — abstract features that are not directly visible in the raw input.
The output layer produces the network's final answer. For a binary classifier (spam or not spam), the output layer might have a single neuron. For a digit recognizer (0-9), it might have ten neurons, one per digit.

Every neuron in one layer is connected to every neuron in the next layer. Each connection has its own weight. A network with 100 input neurons, one hidden layer of 50 neurons, and 10 output neurons has 100 × 50 + 50 × 10 = 5,500 weights to learn. Larger networks have millions or billions.

A simple three-layer network. Every neuron in each layer connects to every neuron in the next. Each connection carries a weight.

How Training Works

An untrained neural network starts with random weights. It will produce essentially random outputs. Training is the process of adjusting those weights, guided by labeled examples, until the network's outputs match the correct answers reliably.

The training cycle repeats for every example in the training set, typically many times over:

Forward pass: Feed an input through the network, layer by layer. Each neuron computes its weighted sum and produces a 0 or 1. The output layer produces a prediction.
Measure the error: Compare the prediction to the correct label. If the network predicted 0 and the correct answer was 1, there is an error. If it predicted 1 and the answer was 1, there is no error.
Adjust the weights: Working backward through the network, slightly adjust the weights that contributed to the error. Weights on connections that led to a wrong prediction are nudged in a direction that would make the correct prediction more likely next time.
Repeat with the next training example.

This process — forward pass, measure error, adjust weights, repeat — is training. After seeing enough labeled examples with enough repetitions, the weights settle into values that produce correct predictions reliably. The network has learned.

What changes and what does not: During training, the weights change. The network's architecture — how many layers, how many neurons per layer, how they are connected — stays fixed. The weights are the knowledge. The architecture is the structure that holds it.

The algorithm that makes backward weight adjustment practical in deep networks is called backpropagation. The mathematical details are beyond our scope here, but the intuition is important: backpropagation efficiently calculates how much each weight contributed to the overall prediction error, so all weights in the network can be adjusted simultaneously in a single backward pass. Without backpropagation, training a network with multiple hidden layers would be impractically slow.

ALVINN: A Real Neural Network

In the mid-1980s, researchers at Carnegie Mellon University built ALVINN — Autonomous Land Vehicle in a Neural Net — a system that learned to steer a van by watching a human driver. It is one of the earliest and most striking demonstrations that a neural network could learn a complex real-world skill from examples.

ALVINN's architecture was surprisingly simple. Its input layer consisted of a 30 × 32 grid of sensors (960 inputs total), each reporting the brightness of one section of a camera image of the road ahead. These inputs fed into a hidden layer of just four neurons. The four hidden neurons connected to an output layer of thirty neurons, arranged to represent steering directions: neurons at one end of the row represented sharp left turns, neurons at the other end represented sharp right turns, and neurons in the middle represented driving straight.

To train ALVINN, researchers had a human driver take the wheel while the network rode along, making its own steering predictions at every moment. After each prediction, ALVINN compared its predicted steering direction to the human driver's actual direction and adjusted its weights slightly to bring its prediction closer to the human's choice. This happened continuously, hundreds of times per second, as the van drove down real roads.

The result: after sufficient training, ALVINN could steer the van at 70 miles per hour. It had learned to drive — not by following rules anyone wrote, but by adjusting 960 × 4 + 4 × 30 = 3,960 weights until they encoded the skill of keeping a vehicle in its lane.

An important lesson from ALVINN: The system learned to steer from normal driving but never learned to recover from mistakes — because the human never made mistakes during training. When the training data was artificially enriched with recovery situations, ALVINN learned those too. This is a direct illustration of a principle that matters for every supervised learning system: the model can only learn what the training data contains. Gaps in the training data become gaps in the learned behavior.

What Comes Next

ALVINN had one hidden layer of four neurons. Modern neural networks used for image recognition, language understanding, and medical diagnosis have tens or hundreds of hidden layers and millions of connections. Reading 3 looks at what changes when networks get deep — and why those changes matter so much for what AI can do today.