Starting With Biology
The human brain contains roughly 100 billion neurons. Each one is a single cell that receives signals from other neurons, combines those signals in a particular way, and either fires — sending its own signal forward — or stays quiet. Individually, each neuron is doing something extremely simple. Collectively, 100 billion of them produce everything from walking to language to the experience of reading this sentence.
A biological neuron has two kinds of structures: dendrites, which are the input fibers that receive signals from neighboring neurons, and an axon, which is the output fiber that carries the neuron's signal to the next neuron. The connection between one neuron's axon and another's dendrite is called a synapse. The conductivity of a synapse — how strongly a signal passes through it — is controlled by its chemical composition. This is where learning happens in biological systems: when we learn something, the strengths of synaptic connections change.
An artificial neural network is a computational model inspired by this biological architecture. It does not replicate the full complexity of a brain — not even close. But it captures the essential idea: simple processing units, connected in layers, where learning happens by adjusting the strength of connections.
The Perceptron: One Artificial Neuron
The simplest unit in an artificial neural network is called a perceptron. A perceptron takes multiple inputs, combines them, and produces a single output of 0 or 1. Its internal mechanics mirror the biological neuron in a simplified form.
Here is what happens inside a perceptron:
- Each input arrives with an associated weight. The weight determines how much influence that input has on the perceptron's decision. A large positive weight means the input strongly encourages a 1 output. A large negative weight means the input strongly discourages a 1 output. A weight near zero means the input barely matters.
- The perceptron computes a weighted sum: each input value is multiplied by its weight, and all the products are added together. If the inputs are v1, v2, v3 and the weights are w1, w2, w3, the weighted sum is v1w1 + v2w2 + v3w3.
- The weighted sum is compared to the perceptron's threshold. If the sum exceeds the threshold, the perceptron fires and outputs 1. If not, it stays quiet and outputs 0.
A Worked Calculation
Let us trace through a concrete example. Consider a perceptron with:
- Three inputs with weights: w1 = −2, w2 = 3, w3 = −1
- Threshold value: 1.5
Input pattern 1: v1 = 1, v2 = 1, v3 = 0
| Input | Value | Weight | Product |
|---|---|---|---|
| v₁ | 1 | −2 | −2 |
| v₂ | 1 | 3 | 3 |
| v₃ | 0 | −1 | 0 |
| Weighted sum | 1 | ||
Weighted sum = 1. Threshold = 1.5. Since 1 < 1.5, the perceptron outputs 0.
Input pattern 2: v1 = 0, v2 = 1, v3 = 1
| Input | Value | Weight | Product |
|---|---|---|---|
| v₁ | 0 | −2 | 0 |
| v₂ | 1 | 3 | 3 |
| v₃ | 1 | −1 | −1 |
| Weighted sum | 2 | ||
Weighted sum = 2. Threshold = 1.5. Since 2 > 1.5, the perceptron outputs 1.
Notice what the weights encode: input v2 (weight +3) strongly encourages a 1 output. Inputs v1 and v3 (negative weights) work against firing. The perceptron is essentially implementing a rule: fire if v2 is present, unless v1 is also present and strong enough to cancel it out.
Weights as Encoded Knowledge
The weights are where the knowledge lives. A perceptron with hand-chosen weights can be programmed to implement specific logical rules. But the real power of neural networks is that the weights do not have to be hand-chosen — they can be learned from data.
Think about what a spam-detecting perceptron might learn. An input representing "message contains 'FREE MONEY'" might get a large positive weight. An input representing "sender is in the contact list" might get a large negative weight — because known contacts rarely send spam. An input representing "message contains an image attachment" might get a small positive weight — a weak signal, but worth noting.
None of those weights were written by a programmer. They emerged from training on thousands of labeled examples. The perceptron learned which signals matter and how much — and encoded that knowledge in its weights.
What One Perceptron Cannot Do
A single perceptron is a linear classifier. It can draw one straight boundary line through the input space and put everything on one side into category 1 and everything on the other side into category 0. That is surprisingly useful for simple problems.
But many real problems are not linearly separable. Recognizing a face requires detecting combinations of features — eyes, nose, mouth — and their spatial relationships. Understanding a sentence requires tracking meaning across many words. No single straight line through the input space can capture those relationships.
This is why neural networks connect many perceptrons together in layers. The output of one layer becomes the input of the next, allowing the network to build up increasingly complex representations of the input. A shallow network might detect edges. A deeper layer might combine edges into shapes. A deeper layer still might recognize that a particular combination of shapes is a face.
In Reading 2, we look at how those layers are connected, and how the entire network is trained together as a unified system.