Data Quality, Analysis and Reporting

What data quality means

Data is of good quality when it is accurate (close to the truth), complete (nothing missing without a reason), consistent (same units and meaning over time), timely (the time stamps are right) and traceable (you can show where each number came from). Quality is a property of the whole chain in this guide, and it is easier to build in than to repair afterwards. The analysis code that reads the files is in Data, Files and Parsing.

Quality	Question	Check
Accuracy	Is the value right?	Calibration, a reference, a second sensor (see Sensors, Signal Conditioning and ADCs)
Completeness	Is anything missing?	Count the samples against the expected number
Plausibility	Is it physically possible?	Range and rate-of-change limits
Timeliness	Is the time right and in order?	Monotonic times, clock sync flags (see Time Stamps and Clocks)
Consistency	Same units and meaning?	Metadata, version of the firmware and scaling
Provenance	Where did it come from?	Logger ID, file, checksum, change log

Validating at the logger and at the server

Check	Catches	Example rule
Range	Open or shorted sensor, wrong units	A cold-room temperature must lie between −40 and 60 °C; a 4–20 mA reading below 3.6 mA is a fault (see Sensors, Signal Conditioning and ADCs)
Rate of change	Spikes, glitches, loose wire	Water temperature cannot move 10 °C in a second
Stuck value	Frozen sensor, dead channel, a repeating last value	The same exact value for 30 minutes on a noisy channel
Cross-check	A drifting or failed sensor	Two probes in the same place within 1 °C; flow in equals flow out
Sequence / counter	Lost or duplicated records	The counter increases by exactly one
Time	Clock jumps, reordering	Time never goes backwards; the interval is within 10 % of nominal
Checksum / CRC	Corrupted records	Computed per record or block (see Storage, Formats and Data Rates)

• Flag, do not delete. Keep the original value and add a quality flag (good, suspect, bad, estimated, missing). Whoever uses the data can then decide.
• Validate twice: simple checks at the logger so that an alarm can fire, thorough ones at the server where you can compare channels.
• Record the test results, and the version of the rules, so that a changed rule does not silently change old results.

import pandas as pd
df = pd.read_csv("log.csv", parse_dates=["t"]).set_index("t").sort_index()
bad_range = ~df["air_c"].between(-40, 60)
bad_rate  = df["air_c"].diff().abs() > 5          # more than 5 degC between samples
stuck     = df["air_c"].rolling(30).std() == 0    # no variation for 30 samples
df["flag"] = (bad_range | bad_rate | stuck).astype(int)

Gaps, duplicates and ordering

• Duplicates come from retries and overlapping downloads: remove them on time + logger ID, keeping the first.
• Out-of-order records come from buffering: sort on the device time, not the file order.
• Overlaps and restarts: when a logger is restarted, the new file starts again; mark the boundary and treat the first samples after a restart with suspicion (sensors warming up).

Outliers and spikes

A single wild value may be a real event, a glitch in the wiring, or a transmission error. Decide by cause, not by looks. A robust test uses the median and the median absolute deviation (MAD), because the mean and standard deviation are themselves pulled by the outlier.

Calibration and correction

Compare the channel with a better reference at two or more points and fit a straight line reference = (reading − offset) ÷ gain. Apply the correction in the analysis and store the coefficients in the metadata, so that the raw data is never overwritten (see Planning a Logging System).

Summary statistics and uncertainty

Statistic	Use	Watch out
Mean	The central value of a steady signal	Pulled by outliers; meaningless for angles or cyclic values
Median	Robust central value	Ignores the extremes
Standard deviation	Spread (noise or variation)	Use n−1 for a sample; not for skewed data
Min / max	Extremes and compliance	Depends on the sample rate and the filter
Percentiles (5, 50, 95)	The typical range, without outliers	Need enough samples
Time above / below a limit	Compliance and excursions	State the limit, the interval and how gaps are counted
Integral (energy, volume)	Totals from rates	Time step and gaps matter: use the totaliser if you have one

Control charts

A control chart separates normal variation from a real change. Take the mean of each subgroup (for example five readings), compute the center line and the standard deviation of the means from a period when the process was known to be stable, and draw limits at ±3σ. A point outside, or a run of points on one side, signals that something changed.

Honest plots

Do	Avoid
Show gaps as breaks in the line	Joining across a missing hour
Start the axis where it is honest, or say that it is cut	A truncated axis that turns 0.2 degrees into a cliff
Plot the right time (UTC or stated local time) and label the axis with units	Unlabeled axes and mixed zones
Use min and max bands with the mean for aggregated data	The mean alone for a fast signal
Keep the same scale across panels you compare	Auto-scaling each panel so a noise floor looks like a trend
Mark events, set-points and limits	A line that nobody can interpret
Show the uncertainty: error bars or a band	False precision on the plot or the table

Reporting and traceability

• State the question and the answer first; the plots and tables support it.
• Say what was measured, how and by whom: sensor, logger, firmware, sample rate, filtering, calibration date, time source (the metadata, see Planning a Logging System).
• Give the completeness and the treatment of gaps and outliers (“3 samples flagged and excluded, 780 s missing in one gap”).
• Quote numbers with their uncertainty and units, to a sensible number of digits.
• Keep the raw data, the code and the report together, with a checksum of the raw files, so the result can be reproduced.
• Regulated records (food safety, pharmaceutical) may require audit trails, electronic signatures, defined retention and validated software; check the rules that apply before you design the system.

Troubleshooting table

Symptom	Likely cause	Check
Readings flat at an odd value	Open input floating, sensor disconnected, ADC clipping	Look at the raw counts; short the input; check the NAMUR level
Noise much larger than expected	Pick-up, ground loop, supply noise	Short the input; shield and ground (see Sensors, Signal Conditioning and ADCs)
A slow oscillation nobody expected	Aliasing of a faster signal	Change the sample rate and see if it moves (see Sampling, Aliasing and Filtering)
Slow drift	Sensor aging, reference drift, temperature	Compare with a reference over time; log the board temperature
Spikes	Switching noise, loose connection, radio interference	Do they coincide with a motor or relay? Do other channels see them?
Steps in the data	Range switching, a restart, a calibration change	Compare with the event log and the metadata
Missing records	Buffer overflow, power loss, full card	Count records, check the sequence number and the free space
Time jumps	Clock step, DST, lost power with no RTC	Look at the sync log (see Time Stamps and Clocks)
Values too high or low by a constant factor	Wrong scaling or units	Compare with the metadata; check against a known point
Files will not open	Corrupt file, power cut during write	Check the CRC; recover to the last good record (see Storage, Formats and Data Rates)

Related

• Data Logging: the overview of this guide.
• Planning a Logging System: the plan that this page tests.
• Sensors, Signal Conditioning and ADCs and Sampling, Aliasing and Filtering: where most quality problems begin.
• Data, Files and Parsing and Numerical Methods: reading, fitting and filtering logs.
• Machine Learning: anomaly detection on logged data.

FAQ

Should I delete bad data?

No: flag it and keep the original. Delete only copies for a particular analysis, and record what was excluded and why.

Can I fill gaps by interpolation?

For a plot or a calculation that needs continuous data, yes, as a flagged estimate and for short gaps. Never present interpolated values as measurements.

How do I know my sensor has drifted?

Compare it with a reference or a second sensor at intervals, and log the difference. A trend in the difference is drift; a step is a fault or a change.

How many significant figures should I report?

Few enough that the last one is meaningful given the uncertainty: if the expanded uncertainty is ±0.6, report 4.3 ± 0.6, not 4.2983.

References

• JCGM 100:2008 (GUM) and JCGM 200 (VIM); ISO 5725 (accuracy of measurement methods).
• Montgomery, D. C., Introduction to Statistical Quality Control; Wheeler, D. J., Understanding Variation.
• Tufte, E., The Visual Display of Quantitative Information; ALCOA+ principles for data integrity (FDA and MHRA guidance).

Last reviewed: October 2026. The numbers on this page are computed from the example data shown; replace them with your own.