Data Preparation, Splitting and Leakage

What good data looks like

A model learns whatever is in its data, including the mistakes. The data should be representative (cover the loads, speeds, temperatures, product types and faults the model will meet), correctly labeled, consistent (same units and meaning throughout) and big enough for the model you choose. A model trained on summer data may fail in winter; one trained on a single machine may not transfer to its twin. Collecting and timing the data properly is the subject of Data Logging, and its page on data quality applies directly here.

Quality	Question	Typical problem
Coverage	Does it include every condition of use?	Only normal running, no faults and no start-ups
Label quality	Are the answers right and consistent?	Two inspectors disagree; labels written weeks later
Balance	Are the categories represented?	1 % faults in the data
Independence	Are the samples different enough?	Consecutive samples nearly identical, so 10 000 rows are really 20
Timing and units	Are channels aligned and in the same units?	Clocks out by seconds, a mix of psi and bar
Provenance	Do you know where each row came from?	Data merged from several machines with no machine ID

Cleaning

• Plot everything first. Most data problems (a stuck sensor, a unit change, a flat line) are obvious in a picture and invisible in a table.
• Fix units and scaling and keep a record of every conversion.
• Handle gaps deliberately. Options: drop the rows, fill from the last value, interpolate (short gaps only) or add a “missing” flag as an input. Dropping 4 % of 10 000 rows loses 400 rows, which matters little if they are scattered but a great deal if they are all one fault.
• Treat sensor faults as faults, not data: a 4–20 mA reading below 3.6 mA is a broken wire, not a low pressure (see the NAMUR note in Sensors, Signal Conditioning and ADCs).
• Be careful with outliers. Removing the rare extreme values may remove the very events you want to detect; flag them and decide by cause.
• Remove duplicates and records written twice by a retry, or they will appear on both sides of a split.
• Align time stamps across channels before building features (see Time Stamps and Clocks).

import pandas as pd
df = pd.read_csv("run.csv", parse_dates=["t"]).set_index("t").sort_index()
df = df[~df.index.duplicated(keep="first")]                  # duplicates
df = df[df["current_ma"] >= 3.6]                              # broken-wire readings are not data
df = df.interpolate(limit=3)                                  # fill gaps of up to 3 samples only
df["missing"] = df["temp_c"].isna().astype(int)               # keep a flag for what is left

Scaling features

Many methods (nearest neighbors, support vector machines, neural networks, regularized regression) treat all inputs alike, so a feature measured in thousands drowns one measured in tenths. Scale them to comparable ranges. Tree-based methods do not need it.

z = (x − μ) ÷ σ scaled = (x − min) ÷ (max − min)

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = make_pipeline(StandardScaler(), LogisticRegression())   # the scaler is fitted on training data only
model.fit(X_train, y_train)

Rare events and class imbalance

Faults are rare, so the “faulty” class is often a tiny fraction of the data. A model can then score very high accuracy by never predicting a fault.

• Use metrics that see the rare class: precision, recall, F1 and the precision-recall curve (see Classification and Metrics).
• Stratify the split so that each part has the same fraction of faults (stratify=y).
• Reweight the classes (class_weight="balanced") so that a fault counts about 99 times as much, or oversample the rare class in the training data only.
• Treat it as anomaly detection when faults are too rare or too varied to learn (see Trees, Clustering and Anomaly Detection).
• Get more faults: run-to-failure tests, seeded faults on a test rig, simulation, or data from similar machines. Label them carefully.

Training, validation and test sets

Set	Purpose	Typical share	Rows of 4 200
Training	Fit the parameters	About 70 %	2 940
Validation	Choose the model, the settings and the threshold	About 15 %	630
Test	One honest final score, used once	About 15 %	630

• Split before you look and before you scale. Whatever you learn from the test set tunes the model to it and spoils the score.
• Keep the test set locked away until the end. If you pick a model because of its test score, the test set has become a validation set.
• Split by the unit that will be new in service: by machine, by batch, by day, not by row. Rows from the same machine in both training and test make the model look better than it will be on a new machine.
• Small data: use cross-validation (below) for choosing, and keep a small test set for the final number.

from sklearn.model_selection import train_test_split, GroupShuffleSplit
# Random rows (only when rows are independent), keeping the fault fraction the same
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.15, stratify=y, random_state=0)
# Keep each machine entirely in one part
gss = GroupShuffleSplit(n_splits=1, test_size=0.15, random_state=0)
tr_idx, te_idx = next(gss.split(X, y, groups=machine_id))

Time-ordered data

When rows follow one another in time, shuffling them is wrong: neighbors are nearly identical, so a random split puts a sample in training right next to its twin in the test set, and the future leaks into the past. Split by time: train on the earlier data, validate on the next stretch, test on the newest, which is exactly how the model will be used.

Cross-validation

k-fold cross-validation splits the training data into k parts, trains on k − 1 of them and validates on the remaining one, k times over, then averages. It gives a steadier estimate than one split, which matters when data is scarce.

• Use the right variant: stratified k-fold for imbalanced classes, group k-fold when rows belong to machines or batches, and a time-series split (expanding window) for ordered data.
• Put all preprocessing inside the pipeline so that scaling and feature selection are re-fitted within each fold, not before it.

Data leakage

Leakage is when information about the answer reaches the model through a route that will not exist in service. It produces impressive test scores and useless real-world performance. It is the most common reason a model “works” in the lab and fails in the field.

Leak	Example	How to avoid
Future information	A maintenance-log field written after a failure; a feature computed with a centered rolling window that uses later samples	Use only what was known at the time of prediction; use trailing windows
Duplicates across the split	The same run or overlapping windows in training and test	Split by run or machine; remove duplicates first
Preprocessing before the split	Scaling or choosing features with all the data	Fit everything on training only, inside a pipeline
A proxy for the label	A column that is only filled in for faulty cases, a file name or an operator ID that correlates with the outcome	Inspect feature importances; remove anything that will not exist in service
Tuning on the test set	Trying ten models and reporting the best test score	Choose on validation; test once

How much data is enough?

Model	Rough need	Remarks
Straight-line fit	Tens of points	Fewer if the range is wide and the noise small
Linear model with p inputs	About 10 rows per parameter	For 12 parameters, about 120 independent rows
Random forest or boosted trees on a table	Hundreds to thousands	More if the classes are imbalanced
Neural network on signals or images	Thousands to millions	Fewer with a pretrained network or augmentation (see Neural Networks and Deep Learning)

• Independent rows count, not total rows: ten thousand windows from one machine on one day are one example of that day.
• Data augmentation (adding noise, shifting, scaling, small rotations for images) stretches a small set, if the changes mimic real variation.
• Learning curves (score against training size, see Overfitting, Validation and Tuning) tell you whether more data would help.

Related

• Machine Learning: the overview of this guide.
• What Machine Learning Is and When to Use It: the workflow this page is part of.
• Overfitting, Validation and Tuning: using the splits to detect overfitting.
• Data Logging and its data quality page.
• Data, Files and Parsing: loading and cleaning in Python.

FAQ

Why should I split before scaling?

Because the scaling numbers (the mean and standard deviation) are learned from data. If they come from the whole set, information about the test rows has leaked into the training. Fit the scaler on the training part only, then apply it to the rest.

Should I shuffle time series?

No. Shuffling puts near-identical neighbors on both sides of the split and lets the future teach the past. Split by time and, where possible, by machine.

How do I handle missing values?

Understand why they are missing. Drop a few scattered rows, fill short gaps by interpolation, and add a “missing” flag for the rest. Never fill the test set using statistics from itself.

What if I have very few faults?

Use stratified splits and recall, precision and F1; reweight the classes; get more fault data by tests or simulation; or switch to anomaly detection trained on normal data.

References

• Géron, A., Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow, O’Reilly (data preparation and validation chapters).
• scikit-learn user guide: cross-validation, preprocessing and pipelines; Kaufman, S. and others, “Leakage in data mining”, ACM TKDD, 2012.
• Zinkevich, M., Rules of Machine Learning, Google (free online).

Last reviewed: October 2026. Counts and scores in the worked examples are illustrative: compute your own.