Classification and Metrics

Logistic regression, thresholds, the confusion matrix, precision, recall, F1, ROC and AUC, and choosing a metric from the cost of errors.

On this page

  1. What a classifier outputs
  2. Logistic regression
  3. The confusion matrix
  4. Metrics and what they hide
  5. Choosing the threshold
  6. ROC curves and AUC
  7. More than two classes
  8. Rare events and the base rate
  9. FAQ

What a classifier outputs

A classifier puts each input into a category: healthy or faulty, pass or fail, which of five fault types. Most classifiers actually produce a score or a probability for each class, and a threshold turns that score into a decision. Moving the threshold trades one kind of error against the other, which is why the choice of threshold and metric matters as much as the choice of model.

OutputMeaningUse
Class labelThe final decision (healthy, faulty)Alarms, sorting, rejecting parts
Score / probabilityHow strongly the model believes a class, between 0 and 1Choose the threshold, rank cases, send the doubtful ones to a person
Per-class probabilitiesOne probability per class, adding to 1Multi-class decisions and “not sure” cases

Logistic regression

Despite its name, logistic regression is the standard simple classifier. It passes a straight-line score through the S-shaped sigmoid so that the output is a probability: p = 1 ÷ (1 + e−(b0 + b1 x)). Like a straight line in regression, it is fast, explainable and the right baseline.

p = 1 ÷ (1 + e−z), z = b0 + b1 x odds = p ÷ (1 − p) = ez

Worked example: fault probability from vibration

Vibration RMS (mm/s) was recorded on 24 machines, 10 of which were later found faulty. Fitting a logistic regression gives b0 = -9.38 and b1 = 3.35 per mm/s.

  • The model says 50 % at x = −b0 ÷ b1 = 2.80 mm/s.
  • At 3.0 mm/s the probability of a fault is 0.66 (odds 1.98 to 1).
  • Each extra 1 mm/s multiplies the odds by e3.35 = 28.6: a clear, checkable statement about what the model learned.
  • The data overlap (some faulty machines have low vibration and some healthy ones high), so no threshold separates them perfectly. That is the usual situation.
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression().fit(X_train, y_train)          # X_train: features, y_train: 0 or 1
p = clf.predict_proba(X_test)[:, 1]                       # probability of "faulty"
pred = (p >= 0.35).astype(int)                            # choose the threshold yourself; do not accept 0.5 blindly

The confusion matrix

Count how the decisions compare with reality. The four cells are true positives (faults caught), false negatives (faults missed), false positives (false alarms) and true negatives (healthy passed).

80True positive10False negative20False positive890True negativeModel saysfaultyhealthyRealityfaultyhealthyPrecision = 0.800TP ÷ (TP + FP) = 80 ÷ 100Recall = 0.889TP ÷ (TP + FN) = 80 ÷ 90F1 score = 0.8422 TP ÷ (2 TP + FP + FN)Accuracy = 0.970(TP + TN) ÷ all = 970 ÷ 1 000A model that always says “healthy” would score 910 ÷ 1 000 = 91 % accuracy and catch no faults.On rare events, look at precision and recall, not accuracy.
A confusion matrix and the metrics that come from it.

Metrics and what they hide

MetricFormulaQuestion it answersWeakness
PrecisionTP ÷ (TP + FP)When it alarms, how often is it right?Ignores the faults it missed
Recall (sensitivity)TP ÷ (TP + FN)Of all real faults, how many does it catch?Ignores false alarms: always alarming gives recall 1
SpecificityTN ÷ (TN + FP)Of the healthy cases, how many pass?Dominated by the many healthy cases
F1 score2 TP ÷ (2 TP + FP + FN)A balance of precision and recallTreats both errors as equally costly
Accuracy(TP + TN) ÷ allHow often is it right overall?Misleading when classes are unbalanced
Balanced accuracy(recall + specificity) ÷ 2Accuracy with the classes weighted equallyHides precision
Matthews correlation(TP·TN − FP·FN) ÷ √((TP+FP)(TP+FN)(TN+FP)(TN+FN))A single balanced number, from −1 to 1Less intuitive

Worked example: a fault detector on 1 000 cases

Of 1 000 cases, 90 are faulty. The model finds 80, misses 10, raises 20 false alarms and passes 890 healthy cases.

  • Precision = 80 ÷ 100 = 0.800: four in five alarms are real.
  • Recall = 80 ÷ 90 = 0.889: it catches about 89 % of faults.
  • F1 = 0.842; specificity = 0.978; balanced accuracy = 0.933; Matthews = 0.827.
  • Accuracy = 0.970, but a model that never alarms scores 0.910 while catching nothing.

Choosing the threshold

Raising the threshold makes the model more cautious: fewer false alarms and more misses. Lowering it does the opposite. Sweep it on validation data (see Data Preparation, Splitting and Leakage) and read off the trade-off.

Worked example: sweeping the threshold on the vibration model

Applying the logistic model to the 24 machines (10 faulty, 14 healthy):

ThresholdTPFPFNTNPrecisionRecallF1
0.20941100.690.900.78
0.35822120.800.800.80
0.50822120.800.800.80
0.65812130.890.800.84
0.80703141.000.700.82
  • A low threshold catches almost every fault but alarms on healthy machines; a high one is quiet but misses faults.
  • A cost-based threshold. If a false alarm costs $200 and a miss costs $5 000, alarm whenever the fault probability exceeds cFA ÷ (cFA + cmiss) = 0.038, far below 0.5. (This assumes the probabilities are well calibrated.)
  • Check calibration: of the cases given about 30 % fault probability, about 30 % should be faulty. If not, calibrate (Platt scaling, isotonic regression) before using probabilities for decisions.

ROC curves and AUC

The ROC curve plots the true-positive rate (recall) against the false-positive rate as the threshold sweeps from high to low. A model that guesses lies on the diagonal; a better model bows toward the top-left corner. The area under the curve (AUC) is the probability that a random faulty case scores higher than a random healthy one: 0.5 is chance, 1.0 is perfect.

A threshold trades false alarms for missed faults-2024model scorehealthyfaultythreshold 1.0000.50.511t = 2.0t = 1.0t = 0.0t = -1.0false positive ratetrue positive rate (recall)AUC = 0.920
Two score distributions overlap; moving the threshold slides along the ROC curve.

Worked example: two bell-shaped score distributions

  • Healthy cases score around 0 and faulty around 2, each with a spread of 1. At a threshold of 1.0 the false-positive rate is 0.159 and the recall is 0.841.
  • Sweeping the threshold traces the curve; its area is AUC = 0.920 (theory: Φ(2 ÷ √2) = Φ(1.414) = 0.921).
  • AUC compares models regardless of threshold, but you still have to pick a threshold to operate. With rare faults prefer the precision–recall curve, because the ROC curve flatters a model when negatives vastly outnumber positives.

More than two classes

With several classes the confusion matrix has a row per true class and a column per predicted class. Compute precision, recall and F1 for each class, then average them: macro (each class counts equally) or weighted (by the number of cases).

Worked example: three machine states

Rows are the true state and columns the prediction (healthy, imbalance, bearing), over 150 cases:

True classPredicted as (h / i / b)PrecisionRecallF1
healthy48 / 2 / 00.890.960.92
imbalance5 / 40 / 50.870.800.83
bearing1 / 4 / 450.900.900.90
  • Accuracy = (48 + 40 + 45) ÷ 150 = 0.887. Macro-F1 = 0.885; weighted F1 = 0.885.
  • Read the matrix, not just the average: the imbalance class is confused with healthy and bearing alike (5 and 5 cases), so its features need work.

Rare events and the base rate

Worked example: a good detector with poor precision

A detector has 95 % recall and 95 % specificity, apparently excellent. But only 1 % of machine-days are faulty. Out of 10 000 days: 100 faulty, of which 95 are caught; 9 900 healthy, of which 495 trigger false alarms. Precision = 95 ÷ (95 + 495) = 0.16: about five of every six alarms are false.

  • When faults are rare, a tiny false-alarm rate still produces many false alarms, because there are so many healthy cases. Precision depends on the base rate; recall and specificity do not.
  • Design the response to match: a cheap first check (look at a trend) before an expensive one (stop the line), and show the operator the evidence.
  • Report the base rate along with the metrics, and test on data with the real fault frequency.
  • Send the doubtful cases to a person: a band of scores between two thresholds can be marked “inspect” instead of a hard decision.

Related

FAQ

What threshold should I use?

Not 0.5 by default. Choose it on validation data from the costs of a miss and a false alarm: with a miss 25 times costlier than a false alarm, alarm above about 4 % probability.

Precision or recall?

Recall when a miss is costly (safety, bearings), precision when false alarms are costly (stopping a line). F1 balances them if both matter equally.

Why is my accuracy high but the model useless?

Imbalanced classes: always predicting the common class is right most of the time. Look at precision, recall and the confusion matrix.

What does AUC tell me?

How well the model ranks faulty above healthy cases across all thresholds: 0.5 is chance and 1.0 is perfect. It does not choose the operating threshold for you.

References

  • James, G., Witten, D., Hastie, T. and Tibshirani, R., An Introduction to Statistical Learning, chapter 4.
  • Fawcett, T., “An introduction to ROC analysis”, Pattern Recognition Letters, 27 (2006).
  • scikit-learn user guide: model evaluation (classification metrics, ROC, precision-recall, calibration).

Last reviewed: October 2026. Data in the examples are illustrative; use your own validation data to set a threshold.