Chapter 4 · Part 2

Why traffic lights

Now the question everyone actually asks: why traffic lights? And crosswalks, and buses, and fire hydrants, and storefronts? The answer is the exact recipe from last chapter — known control + unknown to label — pointed at a new, hugely valuable kind of data: photos of streets.

Once text was digitized, the unsolved problem worth billions was teaching computers to see — to read house numbers for maps, and to recognize the objects a self-driving car must never miss. So the CAPTCHA quietly became an image-labeling machine.

Scroll through the grid you know so well, with its secret revealed.

The familiar 3×3 grid: 'select all images with traffic lights.'

scroll↓

The same trick, new data

Each grid mixes two kinds of tile:

  • Control tiles — the system already knows whether they contain a light. You clicking them correctly proves you're human (and filters bots).
  • Unknown tiles — fresh street imagery with no label yet. Your selection is a vote; once enough people agree, the tile gets a trusted label.

It's reCAPTCHA's two-word idea exactly, with pixels instead of letters.

Why these objects

The categories aren't random — they're the vocabulary of the road, which is why this ties directly into the self-driving course:

  • Traffic lights, crosswalks, stop signs, buses, cars, bicycles, fire hydrants — precisely the objects an autonomous car's perception system must detect to drive safely.
  • Storefronts and house numbers — for building and improving maps.

Every grid you solve adds a labeled example to a dataset that helps a model learn to recognize that object. You are, a few seconds at a time, a data labeler for computer vision — the same kind of labeled images that train the CNNs from our image course.

A beautiful loop — with a catch

Pause on what we've built: a security test that also generates the exact data needed to teach machines to see. It's almost too elegant. But it contains the seed of its own undoing. If solving these puzzles trains computer vision, then computer vision keeps getting better — including at solving these very puzzles. The gate is teaching its own intruders. Next: the arms race.