Training -- Teaching Machines to Learn

The entire field of deep learning was saved by a paper that was rejected, ignored, and reinvented four separate times. The core idea -- that you can work backwards through a neural network to figure out which weights caused the error -- was first published in a Finnish master's thesis in 1970, buried in obscurity. It was reinvented by an American PhD student in 1974 and ignored again. It took until 1986, when Geoffrey Hinton and colleagues published it in Nature, for the idea to finally stick.

That idea is called backpropagation. Without it, there is no ChatGPT, no image generation, no modern AI. And the story of how neural networks learn is, in large part, the story of this one algorithm.

What Training Actually Means

Training a neural network is a five-step loop, repeated millions of times:

A network starts with random weights -- its predictions are garbage. Through training, it gradually improves, like a student learning from practice problems. The "textbook" is the training data. The "grade" is the loss functionA formula measuring how wrong the model's prediction was -- a single number that captures the gap between what the model predicted and the correct answer.. The "study strategy" is gradient descentAn optimization algorithm that adjusts weights in the direction of steepest error reduction, step by step..

Loss Functions: Your Model's Report Card

A loss function boils down the model's performance to a single number -- how wrong was that prediction? The entire goal of training is to make this number as small as possible.

The choice of loss function defines what "good" means for a given task. Without a clear measure of "how wrong am I?", there's no way to improve. It shapes what the network optimizes for -- and by extension, what the network becomes.

Gradient Descent: Hiking Down a Mountain in Fog

Imagine you're on a mountain in thick fog. You can't see the valley floor. You can only feel the slope beneath your feet. Your strategy: take a step in the steepest downhill direction. Repeat. Eventually you reach the bottom.

That's gradient descent.

The gradient is a vector that answers, for each of the millions of weights: "If I nudge this weight by a tiny amount, how much does the loss change?" It points in the direction of steepest ascent -- so you move in the opposite direction to reduce the loss.

That's the entire update rule. Every modern AI system -- from GPT-3 to image generators to self-driving cars -- is trained with some variant of this formula.

In practice, three flavors exist. Batch gradient descent computes the gradient using all training data -- accurate but slow. Stochastic gradient descent (SGD) uses one random sample at a time -- noisy but fast. Mini-batch (typically 32-256 samples) is the practical sweet spot that balances accuracy and speed. Most modern training uses mini-batch SGD or its descendants.

Backpropagation: The Breakthrough That Changed Everything

Here's the problem gradient descent leaves you with: the network has millions of weights, and they all contributed to the error. Which ones deserve the blame, and how much? This is called the credit assignment problem, and it's the hard part.

BackpropagationA method for computing each weight's contribution to the error by working backwards through the network using the chain rule of calculus. solves it by working backwards. Starting from the output, it propagates the error signal through each layer, using the chain rule of calculus to figure out exactly how much each weight contributed to the final mistake. Then gradient descent uses that information to update every weight proportionally.

Andrej Karpathy spent eight years figuring out the best way to explain this, culminating in his micrograd project: just 94 lines of Python that implement everything needed to train a neural network.

"These 94 lines of code are everything that is needed to train a neural network. Everything else is just efficiency."

-- Andrej Karpathy, on micrograd

The Practical Knobs

Training involves three critical hyperparameters. Getting them right is, as practitioners often say, "part science, part art" -- the "dark arts" of deep learning.

Epoch sets the duration ("how long do we train?"). Batch size determines frequency ("how often do we update?"). Learning rate controls magnitude ("how much do we change?"). Together they determine everything about the practical dynamics of training.

The ImageNet Moment (2012)

Fei-Fei Li saw something in 2006 that almost no one else did. While the entire AI research community was focused on building better algorithms, she focused on data.

"The paradigm shift of the ImageNet thinking is that while a lot of people are paying attention to models, let's pay attention to data. Data will redefine how we think about models."

-- Fei-Fei Li, Quartz interview

Her colleagues were skeptical. Spending years collecting data instead of publishing algorithm papers seemed like career suicide for a junior professor. She did it anyway.

Then in September 2012, something happened that stunned the field. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered a deep convolutional neural network called AlexNet into the annual ImageNet competition. The results:

A 10.8 percentage point gap -- unprecedented. And AlexNet was trained on just 2 NVIDIA GTX 580 GPUs, reportedly in Krizhevsky's bedroom at his parents' house. Training took 5-6 days.

Before AlexNet, the dominant approach was: humans manually design feature extractors (SIFT, HOG), then feed those features to a classifier. After AlexNet: feed raw pixels to a neural network and let it learn everything end-to-end.

By 2017, 29 of 38 competing teams had achieved error rates below 5% -- surpassing estimated human performance (~5.1%). The competition was discontinued, having served its purpose.

The Godfathers and Their Long Bet

Geoffrey Hinton, Yann LeCun, and Yoshua Bengio kept faith in neural networks through decades when the approach was considered a dead end.

"There was a dark period between the mid-90s and early-to-mid-2000s when it was impossible to publish research on neural nets, because the community had lost interest in it."

-- Yann LeCun, Understanding AI

"The three of us all knew that it would be the ultimate answer."

-- Geoffrey Hinton, Neural Buddies

Their persistence was rewarded with the 2018 Turing Award (the "Nobel Prize of Computing"). And in 2024, Hinton went further -- winning the Nobel Prize in Physics alongside John Hopfield for "foundational discoveries and inventions that enable machine learning with artificial neural networks."

"If you believe in something, don't give up on it until you understand why that belief is wrong."

-- Geoffrey Hinton, Nobel Prize speech, 2024

The Cost of Training

Training costs have skyrocketed as models scale:

Notice the outlier: DeepSeek V3 achieved competitive results for $5.6M -- about 30x cheaper than Llama 3.1. Training efficiency matters as much as raw scale. Hardware improvements (H200/B200 GPUs) have also driven a 45% cost reduction for standard model sizes. But frontier models keep getting bigger faster than hardware gets cheaper.