A neuron & a tiny network

In Chapter 8 we said the weights of a filter can be learned rather than designed. But learned how, and by what? The answer is a machine built from one tiny, endlessly-repeated part: the artificial neuron.

A neuron does something almost embarrassingly simple. It takes a few numbers in, multiplies each by a weight, adds them up, adds one more number called the bias, and squashes the result into a tidy output. That's it. Stack enough of them and you get a network that can recognise a cat.

Scroll to feed three features into a single neuron and watch it decide.

Weights, bias, activation

Every neuron is the same three-step recipe:

• Weights decide how much each input matters — and the sign decides whether it argues for or against. A big positive weight on "edge" means edges make me more likely to fire.
• Bias shifts the whole decision up or down — a neuron's default leaning before any input arrives.
• Activation is the squash at the end. Without it, a stack of neurons would collapse into one big linear formula; the non-linear kink is what lets networks learn curved, complicated boundaries.

This is the same dot product as before

Look closely at Σ wᵢxᵢ — weights times inputs, summed. That's exactly the operation a convolution kernel does over a patch of pixels in Chapter 5. A neuron and a kernel are the same multiply- and-add machine; the only difference is what they're wired to.

That's the bridge to the next chapter: a convolutional neuron is just a neuron whose inputs are a little window of the image, reused at every position.

import numpy as np

def neuron(x, w, b):
  z = np.dot(w, x) + b      # weighted sum + bias
  return 1 / (1 + np.exp(-z))  # sigmoid activation

x = np.array([0.9, 0.3, 0.6])   # edge, blob, brightness
w = np.array([1.3, -0.9, 0.6])  # learned weights
print(neuron(x, w, -0.2))       # -> a single number in (0, 1)

From one neuron to a network

A single neuron draws one straight line through the data — useful, but limited. The power comes from layers: the outputs of one layer of neurons become the inputs of the next. Early layers detect simple things; later layers combine those into complex things.

We still have two questions open. First: how do all those weights get set? That's training, and it's Chapter 12. Second: how do we wire neurons to images without drowning in 120,000 inputs? That's the convolutional network — next.