How Neural Networks Think

The Artificial Neuron

In 1943, neurophysiologist Warren McCulloch and logician Walter Pitts published "A Logical Calculus of the Ideas Immanent in Nervous Activity" -- the first mathematical model of an artificial neuron. Pitts was 18 years old and had no academic credentials. The paper laid the theoretical foundation for everything that follows in this course.

An artificial neuron does two things. First, it takes all its inputs and computes a weighted sum:

z = w1*x1 + w2*x2 + ... + wn*xn + b

Each input (x) gets multiplied by its weight (w) -- a number representing how much that input matters. The bias (b) is a constant offset, like a baseline mood before hearing anyone's opinion. Then an activation functionA non-linear function that decides whether a neuron "fires" -- without it, the entire network reduces to a single linear equation transforms that sum into the neuron's output.

Think of it this way: each input is a person giving you advice. The weights are how much you trust each person. The bias is your gut feeling before hearing anyone. And the activation function is your decision threshold -- at some point, enough evidence tips you from "no" to "yes."

Layers: Input, Hidden, Output

Individual neurons are simple. The power comes from organizing them into layersA group of neurons that process data at the same level of abstraction -- input layers receive raw data, hidden layers transform it, output layers produce predictions.

The input layer receives raw data and does no computation -- it just passes information forward. For a 28x28 pixel grayscale image (the classic MNIST handwritten digit dataset), that's 784 neurons, one per pixel.

Hidden layers are where the actual "thinking" happens. And here's the key insight: hidden layers discover relevant features automatically, without human engineering. Nobody tells the network "look for edges" or "find circles." It figures that out on its own. The deeper the layer, the more abstract the features:

The output layer produces the final prediction. For digit recognition, it's 10 neurons -- one for each digit 0-9. A softmax function converts the outputs into probabilities that sum to 1: "87% confident this is a 3, 8% it's an 8, 3% it's a 5..."

The Forward Pass

A forward passThe process of feeding input through the network layer by layer to get an output prediction -- no learning happens during this step is what happens every time a neural network processes an input. No learning occurs here -- it's pure prediction. Data flows in one direction: forward.

Here's the step-by-step for recognizing a handwritten digit:

Think of it as a factory assembly line. Raw materials (pixels) enter, each station (layer) refines them, and the final station produces the finished product (a classification). Every worker (neuron) has the same basic job: weigh inputs, sum them, decide whether to activate. The only difference is what they've learned to care about.

Why Non-Linearity Matters

Here's a fact that sounds abstract but is actually the single most important insight in neural network design: without activation functions, your entire network -- no matter how many layers -- is mathematically equivalent to a single linear equation. Stack 100 layers of linear transformations and you get... one linear transformation. Completely useless for learning anything interesting about the real world.

The Universal Approximation Theorem (Cybenko 1989, Hornik et al. 1989) proved the fix: a neural network with just one hidden layer and non-linear activation functions can approximate any continuous function to any desired accuracy, given enough neurons. Non-linearity is what gives neural networks their power.

Two activation functions defined the field:

The Perceptron and Why It Died

In July 1958, the U.S. Office of Naval Research unveiled the Perceptron -- the first machine that could learn from examples. Its creator, Frank Rosenblatt, demonstrated it on an IBM 704 (a 5-ton, room-sized computer). He fed it punch cards, and after just 50 trials, it taught itself to distinguish cards marked on the left from cards marked on the right.

The media response was extraordinary. The New York Times reported that the PerceptronThe first neural network (1958), invented by Frank Rosenblatt. Could only solve linearly separable problems, but proved machines could learn from data. was:

"the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence."
-- New York Times, July 13, 1958

The New Yorker called it "the first serious rival to the human brain ever devised." Rosenblatt himself called it "the first machine which is capable of having an original idea."

Then came Marvin Minsky.

The Rivalry

Minsky was a grade behind Rosenblatt at the Bronx High School of Science in the 1940s. Both traced parallel careers from that same school, but their approaches to AI diverged sharply. Rosenblatt was a connectionist -- he believed in learning from data, bottom-up. Minsky was a symbolist -- he believed in hand-crafted logic and reasoning, top-down.

At conferences, they publicly debated the perceptron's viability, "as their colleagues and students looked on in amazement." Part of the tension was territorial: Rosenblatt's training was in psychology -- "soft science" -- but his perceptron work was mathematical, turf that Minsky, with his Princeton mathematics PhD, didn't feel Rosenblatt belonged on.

The XOR Problem

In 1969, Minsky and Seymour Papert published Perceptrons, a book that proved a devastating mathematical fact: single-layer perceptrons cannot solve the XOR problemExclusive OR: returns 1 only when inputs differ. A single-layer perceptron can't solve it because the outputs aren't linearly separable -- you can't draw one straight line to separate the 1s from the 0s..

XOR (exclusive OR) returns 1 only when its two inputs differ:

If you plot these four points on a graph, there's no single straight line that separates the 1s from the 0s. They sit on opposite corners. A perceptron can only draw one line. For XOR, you need at least two lines -- which means at least two layers.

Minsky and Papert's book was widely misinterpreted as proving that all neural networks were fundamentally limited, not just single-layer ones. It was generally understood at the time that multi-layer networks could solve XOR. But nobody knew how to train them efficiently -- backpropagation hadn't been popularized yet.

The Fallout

The damage was catastrophic. Funding collapsed. The Lighthill Report (1973) declared AI research an "utter failure." DARPA cut grants. Researchers abandoned neural networks for a decade. This was the first AI winter (1974-1980+).

Frank Rosenblatt drowned on his 43rd birthday in 1971 while sailing on Chesapeake Bay. He never lived to see his ideas vindicated. The Perceptron now resides at the Smithsonian Institution. In 2004, IEEE established the Frank Rosenblatt Award in his honor.

As Geoffrey Hinton -- who would eventually lead the neural network revival -- said in his 2024 Nobel Prize speech: "If you believe in something, don't give up on it until you understand why that belief is wrong."

Go Deeper