Overfitting, Validation and Tuning

Fitting the noise

Every measurement is the pattern plus noise. A flexible model can follow the noise in the training data as faithfully as the pattern, scoring brilliantly on data it has seen and poorly on anything new. This is overfitting. The opposite, a model too simple to capture the pattern, is underfitting. The cure is the same discipline throughout this guide: measure the error on data the model has not seen (see Data Preparation, Splitting and Leakage).

Symptom	Diagnosis	Try
Training error low, validation error high	Overfitting (high variance)	More data, a simpler model, regularization, fewer features, early stopping
Both errors high	Underfitting (high bias)	Better features, a more flexible model, less regularization
Both low and close	About right	Confirm on the test set, then deploy
Validation error noisy between runs	Too little validation data	Cross-validation, a larger validation set (see Data Preparation, Splitting and Leakage)

Complexity against error: a computed example

Twenty noisy points follow a smooth curve. Ten (every other one) are used for training, ten for validation. Polynomials of degree 1 to 9 are fitted to the training points.

Degree	Training RMS error	Validation RMS error	Verdict
1	0.63	0.61	underfits
2	0.62	0.65	underfits
3	0.25	0.28	best on validation
4	0.21	0.33	overfits
5	0.16	0.44	overfits
6	0.16	0.38	overfits
7	0.08	1.34	overfits
8	0.07	0.68	overfits
9	0.00	6.02	overfits

Bias and variance

The expected error on new data splits into three parts: bias² (the model is systematically wrong because it is too simple), variance (the fitted model changes a lot with each training set) and the irreducible noise.

expected squared error = bias² + variance + noise²

Learning curves: would more data help?

A learning curve plots training and validation error against the amount of training data. It tells you where to spend effort.

Regularization

Regularization limits a model’s freedom. The common forms: a penalty on the size of the coefficients (ridge: λ Σ w²; lasso: λ Σ |w|), limits on tree depth or leaf size, dropout in neural networks, and early stopping. The penalty λ is a hyperparameter: you choose it by validation, not training.

Tuning without fooling yourself

Hyperparameters (tree depth, k, the penalty, the learning rate, the number of layers) are chosen by trying values and comparing validation scores. Do this with grid search (every combination of a few values), random search (random combinations, which covers important parameters better when few matter) or Bayesian tools such as Optuna. But every comparison on the validation set uses up a little of its honesty.

• Keep the test set out of the loop. If a test score changes any decision, it has become part of validation; get new test data.
• Use nested cross-validation when data is scarce: an inner loop picks the hyperparameters, an outer loop scores the whole procedure.
• Prefer fewer, bigger choices: a handful of values over a wide grid, and the simplest model within one standard error of the best (“one-standard-error rule”).
• Set the random seeds and record them, so that an experiment can be repeated (see What Machine Learning Is and When to Use It).

from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
grid = {"max_depth": [3, 5, 8, None], "min_samples_leaf": [1, 3, 10]}
search = GridSearchCV(RandomForestClassifier(n_estimators=300, random_state=0), grid, scoring="f1",
                      cv=StratifiedKFold(5, shuffle=True, random_state=0)).fit(X_train, y_train)
print(search.best_params_, search.best_score_)     # a validation score: optimistic
print(search.score(X_test, y_test))                # use the test set once, at the very end

Early stopping and other brakes

• Early stopping: for iterative training (boosted trees, neural networks) watch the validation error each round or epoch and stop when it stops improving for a set number of rounds; keep the model from the best round. A common default patience is 5 to 20 epochs.
• Simpler models and fewer features: remove inputs that carry no physical meaning; each extra input is another chance to fit noise.
• Data augmentation: adding realistic variation (noise, shifts, scaling) to training examples makes a model less sensitive to the exact examples.
• Ensembles: averaging many models (forests, bagging) reduces variance (see Trees, Clustering and Anomaly Detection).
• More and better data is the strongest remedy for overfitting; check with a learning curve first.

A validation checklist

Check	Why
A baseline is reported beside the model	A score means nothing without the simple alternative
Splits follow the real use (by machine, by time)	Otherwise the score is too good
Scaling and feature selection are fitted on training data only	No leakage through preprocessing
Validation score uncertainty is stated (cross-validation spread or a margin of error)	Small differences are noise
Training and validation errors are both shown	The gap shows over- or underfitting
The test set was used once	It is the honest number
The error is compared with the noise floor and with the requirement	Better than the noise is a warning; worse than the need is a failure
The model is tried on a new machine, batch or season before launch	The first real deployment test

Related

• Machine Learning: the overview of this guide.
• Data Preparation, Splitting and Leakage: how to split and avoid leakage.
• Neural Networks and Deep Learning: dropout, early stopping and the training curve of a network.
• Deployment, Monitoring and Limits: what happens after validation, when the world changes.

FAQ

How do I know if I am overfitting?

Compare training and validation error. A low training error with a much higher validation error is overfitting. Plot both against model complexity or training size to see it.

How big should the validation set be?

Large enough that the uncertainty of the score is smaller than the differences you want to see. For a classifier near 90 % accuracy, about 600 samples give a margin of about 2.4 points; with rare faults you need enough faults, not just rows.

Is it wrong to look at the test set more than once?

Each look that changes a decision makes the test score optimistic. Use it once, at the end; if you need to look again, collect fresh test data.

Does regularization always help?

It helps when the model is too flexible for the data. If the model is already underfitting, more regularization makes it worse. Choose the strength by validation.

References

• Hastie, T., Tibshirani, R. and Friedman, J., The Elements of Statistical Learning, chapters 7 and 3 (bias-variance and shrinkage).
• Cawley, G. C. and Talbot, N. L. C., “On over-fitting in model selection and subsequent selection bias in performance evaluation”, JMLR, 11 (2010).
• Géron, A., Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow, chapter 4 (learning curves and regularization).

Last reviewed: October 2026. All numbers on this page come from the simulated data described; the pattern, not the values, is what carries over.