Trees, Clustering and Anomaly Detection

Decision trees

A decision tree is a flowchart learned from data: at each node it asks a yes/no question about one feature (“vibration RMS above 2.8 mm/s?”) and sends the case left or right, until a leaf gives the answer. It handles numbers of any scale, mixes of features, and non-linear relationships without scaling, and you can read the rules it learned. The question at each node is chosen to make the groups as pure as possible, measured by the Gini impurity.

Gini = 1 − Σ p_k² (p_k = fraction of class k in the node)

• Strengths: no scaling, mixed feature types, readable rules, fast.
• Weaknesses: a single tree is unstable (small data changes give a different tree), makes steps rather than smooth curves, and cannot extrapolate beyond the training range.

Random forests and boosting

Combining many trees fixes the instability. A random forest grows hundreds of trees, each on a random resample of the rows and a random subset of the features at each split, and averages their votes. Gradient boosting (XGBoost, LightGBM, scikit-learn’s HistGradientBoosting) builds trees one after another, each correcting the errors of the sum so far. On tables of sensor and process data, these are often the strongest models you can build without deep learning.

• Feature importance tells you which inputs the forest uses: use permutation importance (shuffle one feature and see how much the score drops), which is more honest than the built-in measure. An unexpected top feature (a machine ID, a time stamp) is a leakage warning (see Data Preparation, Splitting and Leakage).
• Out-of-bag error (each tree is scored on the rows it did not see) gives a free validation estimate.
• Boosting needs more care (learning rate, number of trees, depth) and overfits more easily than a forest; use early stopping on a validation set.
• Neither extrapolates: outside the training range the prediction stays at the edge value.

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=3, class_weight="balanced", random_state=0, n_jobs=-1).fit(X_train, y_train)
imp = permutation_importance(rf, X_val, y_val, n_repeats=20, random_state=0)
for name, m in sorted(zip(feature_names, imp.importances_mean), key=lambda t: -t[1]): print(name, round(m, 3))

Nearest neighbors

k-nearest neighbors classifies a new case by a vote among the k most similar training cases. There is no training step, which makes it a quick baseline, but it stores all the data, is slow with large sets, and depends entirely on the meaning of “similar”, which means scaling the features.

Clustering: k-means

k-means finds k groups without labels. Start with k centers, assign each point to the nearest, move each center to the mean of its points, and repeat until nothing changes. It suits roundish groups of similar size, and is used to discover operating modes, segment customers or compress data.

• Scale the features first; k-means uses plain distance.
• Results depend on the starting centers: run several starts (n_init) and keep the best; k-means++ initialization helps.
• It cannot find odd shapes (rings, long snakes); try DBSCAN or a Gaussian mixture, which also find noise points.
• Clusters are hypotheses. Check that each one means something physical (a speed, a product, a mode) before acting on it.

Anomaly detection

Often you have plenty of normal data and few or no examples of faults. An anomaly detector learns what normal looks like and flags what does not fit. It cannot say what is wrong, only that today is unlike the past, which is still valuable.

The z-score

Several signals together: the Mahalanobis distance

Isolation forests and autoencoders

An isolation forest builds random trees that split the data at random thresholds; an unusual point is separated from the rest in few splits, a typical point needs many. The anomaly score is 2^{−E(h) ÷ c(n)}, where E(h) is the average path length and c(n) = 2 (ln(n − 1) + 0.5772) − 2 (n − 1) ÷ n is the typical path length for n samples.

An autoencoder is a small neural network trained to squeeze normal data through a narrow layer and rebuild it; faulty data rebuilds badly, and the reconstruction error is the anomaly score (see Neural Networks and Deep Learning). Whatever the method:

• Train on normal data only, and make sure it covers all normal operating modes, seasons and loads.
• Set the alarm threshold on real data, from the false-alarm rate you can afford (see the base-rate example in Classification and Metrics).
• Validate with seeded or recorded faults where possible, even a few, to see what the detector actually catches.
• Plan what happens on an alert: who looks, with what evidence (which channel moved most), and how a false alarm is fed back.

Choosing a method

Situation	Start with	Then
A table of sensor and process numbers, labeled	Logistic regression or linear baseline, then a random forest	Gradient boosting if you need the last few per cent
Few labeled faults, plenty of normal data	Per-mode z-score or Mahalanobis	Isolation forest or an autoencoder
Want to know the operating modes	k-means (scaled), check the elbow	Gaussian mixture or DBSCAN for odd shapes
Little data and a clear relationship	Linear or ridge regression (see Regression and Curve Fitting)	A small tree for readable rules
Images, sound, raw waveforms	A pretrained network, fine-tuned (see Neural Networks and Deep Learning)	Hand-made features plus a forest (see Features for Signals and Machines)
A decision someone must be able to explain	A shallow tree or logistic regression	Feature importance and examples of cases

Related

• Machine Learning: the overview of this guide.
• Classification and Metrics: thresholds and metrics for these models.
• Features for Signals and Machines: the features to feed them for machine data.
• Overfitting, Validation and Tuning: tuning depth, k and penalties without fooling yourself.

FAQ

Should I use a random forest or a neural network on sensor data?

Start with a random forest or gradient boosting on well-chosen features: it needs little tuning, handles mixed scales and works on modest data. Move to a neural network for raw images, audio or very large data sets.

How do I choose k in k-means?

Run several values and look for the elbow in the inertia curve, check the silhouette score, and ask whether each cluster means something physical. There is no single right answer.

Why does my anomaly detector alarm all the time?

Normal was not fully captured (missing operating modes or seasons), the threshold is too tight, or the noise is heavier-tailed than assumed. Add context inputs, use a persistence rule and set the threshold on real data.

Can a tree extrapolate?

No. Beyond the training range a tree or forest keeps returning the value at the edge. Use a model with the right shape, or make sure the data cover the range of use.

References

• Breiman, L., “Random forests”, Machine Learning, 45 (2001); Liu, F. T., Ting, K. M. and Zhou, Z.-H., “Isolation forest”, IEEE ICDM, 2008.
• James, G., Witten, D., Hastie, T. and Tibshirani, R., An Introduction to Statistical Learning, chapters 8 and 12.
• scikit-learn user guide: ensembles, clustering and outlier detection; Chandola, V., Banerjee, A. and Kumar, V., “Anomaly detection: a survey”, ACM Computing Surveys, 2009.

Last reviewed: October 2026. Data in the examples are small and illustrative: use your own with proper validation.