The dwarf planet Ceres, a grey cratered sphere against black space, photographed by NASA's Dawn spacecraft
Unit 04 · All masterclasses · 60 min

Learning from Data

Linear regression, the loss function, gradient descent, and overfitting
Image: NASA/JPL-Caltech/UCLA/MPS/DLR/IDA, image processing by Andrea Luck, CC BY 2.0

Every learning method needs a model with parameters, a loss that measures how wrong it is, and a procedure that lowers the loss. This unit builds all three on a straight line through six house prices, derives least squares, takes the first gradient descent steps on a real loss, and then shows with polynomials why a model that fits its training data perfectly can predict badly on new data, how cross-validation and a ridge penalty control that, and why the largest modern models escape the classical picture.

Preview: the beginning of the unit

Every method in this masterclass that improves with experience does three things, and this unit shows all three on the smallest example that is not a toy: a straight line through the prices of six houses. The line has two numbers that must be chosen, a formula says how wrong a choice is, and a procedure moves the numbers until the formula is as small as it can be. The networks of Units 6 and 7, with millions of parameters, add nothing to this recipe except size, and they share its danger: a model can match the data it was trained on perfectly and still be useless on the next house.

1 Three ingredients of learning

The machines of Units 2 and 3 ran mostly on numbers that a person had written down, such as the length of a road or the probability of a fever given the flu. In machine learning the rules are left open, and the data fills them in. Every learning method in this program, from the line of this unit to the largest language model, consists of the same three parts.

  1. A model, which is a family of functions from inputs to outputs with a set of adjustable numbers called the parameters. A straight line \(\hat{y} = wx + b\) is a model with two parameters, the slope \(w\) and the intercept \(b\). The hat on \(\hat{y}\) marks a prediction of the model, as opposed to a measured value \(y\).
  2. A loss function, which is a single number that says how wrong the model is on the data. It is zero for a perfect fit and grows with every miss.
  3. A learning procedure, which is a method that changes the parameters so that the loss becomes smaller.

The third part is the learning. When someone says that a model “has learned” a task, they mean that the procedure has found parameters with a small loss, and nothing more. Following the notation of Unit 1, all parameters of a model are collected in one vector \(\boldsymbol{\theta}\), which for the line is \(\boldsymbol{\theta} = (w, b)\), and the loss as a function of the parameters is written \(L(\boldsymbol{\theta})\).

The data come as a table. Each row is one example, each column except the last is a feature, and the last column is the target, the value the model should predict. The training set holds \(N\) examples \((\mathbf{x}_i, y_i)\) with \(i = 1, \ldots, N\), where \(\mathbf{x}_i\) is the vector of the features of example \(i\), \(d\) is the number of features, and \(y_i\) is its target. When the target is a number, such as a price or a temperature, the task is called regression, and that is the task of this unit. When the target is a category, such as a species or the word “spam”, the task is classification, which Unit 5 treats with the tools built here.

To illustrate this, the examples of this unit come from the Ames housing data, a public table of 2,930 houses sold in Ames, Iowa, between 2006 and 2010, with 80 columns per house, which the statistician Dean De Cock published in 2011 for teaching. The feature is the floor area of the living space, and the target is the sale price.

1.1 Training set and test set

One rule stands above all others in this field. Before anything is fitted, the rows of the table are split at random into two parts. The training set is the part the learning procedure is allowed to see, and the test set is locked away and used exactly once, at the very end, to measure how the finished model does on examples it has never seen. A common split keeps 80 per cent of the rows for training and 20 per cent for testing.

The reason is that the loss on the training data measures how well the model has memorized those rows, and memorizing is cheap. A model that stores the whole table and looks up every answer has a training loss of zero and knows nothing about the next house. What a buyer wants is a prediction for a house that is not in the table, and only data that played no part in the fitting can measure that. The ability to predict well on new examples is called generalization, and the test set is the only honest estimate of it. A student who has seen the exam questions during the homework would score well for the same bad reason.