PYTHON / MACHINE LEARNING WITH PYTHON
Overfitting, underfitting, and regularisation
Tell underfitting from overfitting by comparing training and held-out error, and use Ridge/Lasso alpha to trade training fit for generalisation.
What you will learn
- Compare training error with held-out error to tell high bias from high variance
- Use the alpha of Ridge or Lasso to trade training fit for better generalisation
- Recognise that a perfect training score can happen on pure noise and proves nothing
- Tune alpha by cross-validation on the training set, never against the test set
Understanding Overfitting, underfitting, and regularisation
Every model has a capacity: the range of functions it can express. Training error falls monotonically as capacity grows, but error on unseen data is U-shaped, dropping while the extra flexibility captures real structure and then rising once the model starts fitting the noise specific to your sample. Underfitting is the left side of that curve (the model cannot express the pattern at all, high bias) and overfitting is the right side (it has memorised your sample's accidents, high variance). Neither is visible from the training score alone, which is why you always score twice: once on the data the fit saw and once on data it did not.
The main example shrinks the effect down to arithmetic you can check by hand. Four training points, the last one noisy: a straight line cannot pass through all four so it leaves a training MSE of 4.80, while a cubic passes through every point exactly and its training MSE is 0.00. Now look just outside the training range at x = 4, where the true value is about 4: the line says 12 and the cubic says 36. The cubic did not learn more about the pattern; it spent its extra freedom honouring one noisy y value, and that distortion is worst near and beyond the edges of the data.
Regularisation lets you keep a flexible model and penalise flexibility rather than remove it. Ridge minimises squared error plus alpha times the sum of squared coefficients, so a large weight is only worth having if it reduces error enough to pay for itself; Lasso penalises the sum of absolute values instead and can drive coefficients exactly to zero, which doubles as feature selection. alpha = 0 reproduces ordinary least squares, and as alpha grows without bound every weight goes to zero so the model just predicts the training mean; useful values lie in between and you find them with cross-validation. Because the penalty is on coefficient size, standardise features first, otherwise "size" just reflects whatever units each column happens to use.
import numpy as np
# Four training points. The underlying pattern is y = x, but the last y is noisy.
x_train = np.array([0.0, 1.0, 2.0, 3.0])
y_train = np.array([0.0, 1.0, 2.0, 11.0])
# One held-out point that follows the true pattern.
x_test, y_test = 4.0, 4.0
for degree in (1, 3):
coeffs = np.polyfit(x_train, y_train, degree)
train_mse = np.mean((y_train - np.polyval(coeffs, x_train)) ** 2)
test_err = (y_test - np.polyval(coeffs, x_test)) ** 2
print(f"degree {degree}: train MSE {train_mse:6.2f} held-out squared error {test_err:8.2f}")Held-out error, not training error, is the quantity you are actually optimising, and regularisation is the continuous dial that moves a model between too rigid and too flexible.
Worked examples
What alpha actually does to a coefficient
Ridge on a single feature shows shrinkage as a closed-form trade between fitting the data and keeping weights small.
import numpy as np
from sklearn.linear_model import Ridge
X = np.array([[1.0], [2.0], [3.0]])
y = np.array([2.0, 4.0, 6.0]) # exactly y = 2x
for alpha in (2.0, 14.0, 126.0):
model = Ridge(alpha=alpha, fit_intercept=False).fit(X, y)
w = model.coef_[0]
mse = np.mean((y - model.predict(X)) ** 2)
print(f"alpha={alpha:5.1f} coef={w:.3f} train MSE={mse:6.2f}")Example explained
Line 1With one feature and no intercept the solution is sum(x*y) / (sum(x*x) + alpha) = 28 / (14 + alpha), which is why the coefficients come out at exactly 1.75, 1.0 and 0.2.
Line 2alpha = 14 exactly matches sum(x*x), so the penalty carries as much weight as the data and the coefficient is halved from its true value of 2.
Line 3Training MSE rises with every increase in alpha: regularisation buys stability by deliberately giving up training fit, and at alpha = 126 it has gone too far and underfits.
Line 4fit_intercept=False keeps the arithmetic transparent; scikit-learn never penalises the intercept, so in real code you leave it on.
A perfect training score on data with no signal
Least squares reaches R2 = 1.0 on random noise as soon as there are more columns than rows.
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(0)
X = rng.normal(size=(6, 20)) # 6 samples, 20 columns of pure noise
y = rng.normal(size=6) # target unrelated to X
model = LinearRegression().fit(X, y)
print(f"samples={X.shape[0]} features={X.shape[1]}")
print(f"train R2 = {model.score(X, y):.3f}")Example explained
Line 1X and y are drawn independently, so there is no relationship at all to learn.
Line 2With 20 columns and only 6 rows the least-squares system is underdetermined: a weight vector that reproduces y exactly always exists, so every residual is zero.
Line 3model.score returns R2, and 1.000 here means the model memorised six random numbers rather than learned anything.
Line 4Score the same model on fresh rows from the same generator and R2 will be near zero or negative, which is the whole reason a train-only score cannot detect overfitting.
Important notes
The direction of the regularisation dial differs by estimator: for Ridge and Lasso, larger alpha means stronger penalty, but for LogisticRegression and SVC the parameter is C and larger C means weaker penalty.
Regularisation cannot fix a model that is too simple. If training and held-out error are both high, raising alpha pushes both higher; you need more capacity or better features instead.
Common mistakes
Judging a model by its training score: an unpruned tree, a 1-nearest-neighbour model and a degree n-1 polynomial all score perfectly on their training data and can still do worse than predicting the mean on new rows.
Picking alpha or tree depth by whichever value gives the best test score. After a handful of attempts the test set has effectively been fitted too, and the number you report is optimistically biased.
Applying Ridge or Lasso to unscaled features: a column in metres and a column in millimetres receive wildly different effective penalties, so which coefficients get shrunk depends on your units rather than on the data.
Try it yourself
Change, predict, then run
Rerun the Ridge example with alphas 0.001, 1, 10, 100 and 1000, and for each one also print the squared error at the held-out point x = 4, y = 8 (the true value under y = 2x). Note which alpha wins and explain why noise-free data prefers almost no regularisation.
Open the Python workspaceCheck your understanding
A Ridge model scores R2 = 0.62 on the training set and 0.61 on the held-out set. What is the most reasonable diagnosis and next move?
- It is underfitting: lower alpha or give it richer features such as interaction terms
- It is overfitting: raise alpha to close the gap between the two scores
- It is already optimal, because the training and held-out scores agree
- The split leaked information: reshuffle and re-split before doing anything else
Show answer
A gap of 0.01 means variance is not the problem, so the ceiling comes from bias: the model family or the current alpha is too restrictive. Raising alpha attacks an overfitting problem that does not exist and would push both scores down, and matching scores only tell you the estimate is trustworthy, not that 0.62 is the best achievable.