Regression and Curve Fitting

Fitting a line

Regression predicts a number from inputs. The simplest model is a straight line y = m x + b. Least squares picks the line that makes the sum of the squared vertical errors (the residuals) as small as possible. For data points (x_i, y_i) with means x̄ and ȳ:

m = Σ (x − x̄)(y − ȳ) ÷ Σ (x − x̄)² b = ȳ − m x̄ R² = 1 − SSE ÷ SST

R² is the fraction of the variation in y that the line explains: 1 is perfect, 0 is no better than the average (the baseline in What Machine Learning Is and When to Use It).

import numpy as np
from sklearn.linear_model import LinearRegression
x = np.array([1, 2, 3, 4, 5]).reshape(-1, 1)
y = np.array([2.1, 3.9, 6.2, 7.8, 10.1])
model = LinearRegression().fit(x, y)
print(model.coef_[0], model.intercept_)     # 1.99 and 0.05
print(model.score(x, y))                    # R squared, about 0.9973
print(model.predict([[6]]))                 # about 11.99

How well is the line known?

The slope and intercept are estimates from a few noisy points; they come with uncertainty. The residual standard deviation is s = √(SSE ÷ (n − 2)), and the standard error of the slope is s ÷ √Σ(x − x̄)².

Several inputs

The same idea extends to many inputs: y = b + m₁ x₁ + m₂ x₂ + …, solved by the same least-squares machinery. Extra inputs let the model explain more, at the cost of more parameters to estimate.

• Include inputs that make physical sense; a coefficient with the wrong sign is a warning that something is confounded.
• Check units and scale before reading coefficient sizes: a bigger coefficient does not mean a more important input unless the inputs are scaled (see Data Preparation, Splitting and Leakage).
• Do not add inputs until R² looks good. Judge on held-out data (see Overfitting, Validation and Tuning).

Residuals: is the model right?

A single number such as R² can hide a model that is wrong in a systematic way. Always plot the residuals (data minus model) against the input and against the prediction. They should look like random scatter around zero with a roughly constant spread.

Pattern in the residuals	Meaning	What to do
A curve (U or arch)	The relationship is not a straight line	Add a squared term or use a nonlinear model
Spread growing with x	The error is proportional to the value	Fit the logarithm, or weight the points
A run of the same sign	Drift over time or a missing input	Plot against time; add the missing variable
One huge residual	A faulty reading or a different regime	Check the cause before removing it
Residuals correlated with each other	Samples are not independent (time series)	Model the dynamics; the standard errors are too small

Curves and the danger of extrapolating

• Trust interpolation, distrust extrapolation: predictions outside the range of the training inputs have no support from the data. Saturation, yield, boiling and limits all bend curves.
• High-order polynomials wiggle between and beyond the points; keep the degree low and test on held-out data (see Overfitting, Validation and Tuning).
• Use physics where you can: a model with the right shape (an exponential, a power law) extrapolates far better than a polynomial.

Exponentials and other curves

Many physical laws become straight lines after a transformation. A cooling curve T = T_amb + ΔT e^−t/τ is exponential, but ln(T − T_amb) = ln ΔT − t ÷ τ is a straight line in t, so least squares finds τ.

Law	Transform to a line	Plot
y = a ekx	ln y = ln a + k x	ln y against x
y = a xn	ln y = ln a + n ln x	ln y against ln x (log–log)
y = a + b ÷ x	y against 1 ÷ x	y against 1 ÷ x
Beta thermistor	1 ÷ T = A + B ln R	1 ÷ T against ln R

Many inputs and regularization

With many inputs, or inputs that nearly duplicate each other, ordinary least squares gives large, unstable coefficients that swing wildly with small changes in the data. Ridge regression adds a penalty λ Σ m² on the coefficients; lasso penalizes Σ |m| and can set some coefficients exactly to zero, which selects inputs. The penalty trades a little bias for a great deal of stability.

from sklearn.linear_model import Ridge, Lasso, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
model = make_pipeline(StandardScaler(), RidgeCV(alphas=[0.01, 0.1, 1, 10, 100]))   # picks the penalty by cross-validation
model.fit(X_train, y_train)

Calibration as regression

Related

• Machine Learning: the overview of this guide.
• Overfitting, Validation and Tuning: judging a fit on data it has not seen.
• Classification and Metrics: when the answer is a category, not a number.
• Numerical Methods: nonlinear least squares and interpolation.
• Sensors, Signal Conditioning and ADCs: where the noise comes from.

FAQ

What is a good R²?

It depends on the noise in the data and what you need. For a calibration you may expect 0.999; for a soft sensor on a messy process 0.7 may be useful. Judge by prediction error on new data, in the units of the output, and by the residual plot.

When should I use a polynomial?

When a plot shows a smooth curve and you have no physical law. Keep the degree low (2 or 3), test on held-out data, and never extrapolate with it.

What is the difference between ridge and lasso?

Ridge shrinks all coefficients smoothly; lasso can set some to zero, selecting inputs. Use ridge for correlated inputs, lasso when you suspect only a few matter.

Should I use a neural network instead?

Try the line, then a tree model, first. A network helps with many inputs and strong non-linearity, but needs more data and is harder to check. See Neural Networks and Deep Learning.

References

• James, G., Witten, D., Hastie, T. and Tibshirani, R., An Introduction to Statistical Learning, chapters 3 and 6.
• Montgomery, D. C., Peck, E. A. and Vining, G. G., Introduction to Linear Regression Analysis, Wiley.
• The scikit-learn user guide: linear models; scipy.optimize.curve_fit documentation.

Last reviewed: October 2026. Data in the worked examples are illustrative: the method is what to reuse.