Deployment, Monitoring and Limits
Models on microcontrollers, quantization and memory budgets, drift monitoring, retraining, safety limits and a troubleshooting table.
Where the model runs
Training happens on a PC or in the cloud. Where the trained model then runs (inference) is a design choice with consequences for latency, cost, privacy and reliability.
| Place | Strengths | Limits |
|---|---|---|
| Cloud or server | Large models, easy updates, central data | Needs a connection; latency; data leaves the site; running cost |
| Edge computer (Raspberry Pi, Jetson, industrial PC, gateway) | Real models, on site, GPU optional, Python | More power and cost; an operating system to maintain |
| Microcontroller (“TinyML”) | Milliwatts, instant response, no network, cheap | Kilobytes of memory; small models only; a harder toolchain |
| PLC or drive | Next to the control, familiar to plant staff | Few vendors support models; fixed function blocks; certification |
“TinyML” tools (TensorFlow Lite for Microcontrollers, Edge Impulse, STM32Cube.AI, microTVM) convert a trained model into C arrays and a small interpreter. See Embedded Systems for the platform side (memory, timers, power).
Will it fit? Memory
Two kinds of memory matter. Flash holds the program and the model’s weights (read-only at run time). SRAM holds the working memory: the intermediate results of each layer (the activations or tensor arena), the input buffer and the stack. Both must fit.
Worked example: 250 000 parameters
- As 32-bit floats: 250 000 × 4 bytes = 1.00 MB. As 8-bit integers: 250 000 × 1 byte = 0.25 MB.
- A small microcontroller with 256 kB of flash cannot hold even the 8-bit model along with its program (244 kB of weights leaves almost nothing). One with 1 MB of flash holds it comfortably; the float model fits in 1 MB only barely, with no room for the program.
- The activations, say 96 kB, must fit in SRAM as well: too much for the 64 kB part, fine for the 256 kB part.
- The SRAM arena is often the real limit. Reduce it with smaller inputs, fewer channels in early layers, stride instead of pooling, and tools that report the peak arena size.
Quantization
Quantization stores weights (and activations) as 8-bit integers instead of 32-bit floats: a quarter of the size, faster on chips without a floating-point unit, and lower in energy. A real value is mapped to an integer by a scale factor: q = round(w ÷ scale), and recovered as q × scale. Post-training quantization converts a finished model using a few hundred sample inputs to set the scales; quantization-aware training simulates it during training for better accuracy.
Worked example: quantizing five weights
The largest weight magnitude is 0.93, so the scale is 0.93 ÷ 127 = 0.00732 per integer step.
| Weight | Integer (−127 to 127) | Recovered | Error |
|---|---|---|---|
| -0.82 | -112 | -0.8202 | -0.0002 |
| 0.05 | 7 | 0.0513 | 0.0013 |
| 0.41 | 56 | 0.4101 | 0.0001 |
| 0.93 | 127 | 0.9300 | 0.0000 |
| -0.27 | -37 | -0.2709 | -0.0009 |
- The largest error is 0.0013, at most half a step (0.0037). Over thousands of weights, these errors average out and the accuracy usually drops by less than a percentage point.
- Outliers hurt: one very large weight makes the scale coarse for all the others; per-channel scales and clipping help.
- Always re-measure the accuracy of the quantized model on the test set, and on the device itself: the results differ slightly from the float model.
Speed and energy
Worked example: one inference on a Cortex-M4 at 80 MHz
- The dense network does about one multiply-accumulate per parameter: 250 000 MACs.
- Assume a conservative 0.5 MAC per cycle (the Cortex-M4’s DSP instructions can do better with optimized kernels), and layer overhead adds about 50.0 %: 250 000 × 1.5 ÷ (80 MHz × 0.5) = 9.4 ms.
- At 20 mA and 3.3 V that is 0.62 mJ per inference. Running one inference a second keeps the processor busy 0.9 % of the time, for an average current of about 0.19 mA including sleep: a 1 000 mAh cell lasts about 217 days.
- Do not forget the feature extraction. A 512-point FFT is about 5 N log2 N = 23 040 operations, perhaps 1.2 ms here: comparable with the network.
- These are estimates. Measure on the real chip with a timer and a current probe; compilers, cache and memory wait states change the numbers.
The whole pipeline on the device
- The sampling, filtering, windowing and feature code are part of the model. They must match training exactly: the same window length, the same filter, the same scaling constants, the same units. A different filter order is a different model.
- Write one reference implementation (in C, shared between the PC and the device) or test the device code against the Python one on recorded windows and require matching outputs.
- Fixed-point and float differences change the numbers a little; check on the test set that the decisions hold.
- Budget the real-time constraints: sample without gaps (DMA), process in the gaps, and make sure the worst-case inference time fits in the window (see Embedded Systems and Time Stamps and Clocks).
- Keep a way out: a watchdog, a safe default if the model fails or the input is out of range, and a flag in the data when the fallback was used.
Drift: monitoring a model in service
The world changes after deployment: bearings wear, sensors age and drift, materials and operators change, a competitor’s product shows a new pattern. The model, frozen at training time, slowly gets worse. Drift comes in two forms: data drift (the inputs change) and concept drift (the relationship between inputs and answer changes). You monitor both.
Worked example: the error rises
- For the first 18 weeks the mean absolute error is 0.80. From week 18 the machine’s behavior changes and the error climbs; it first crosses the alert limit of 1.5 in week 26. Weeks 25 to 30 average 1.64.
- The model is retrained on recent data in week 30, and the error returns to about 0.82.
- You can only plot this if you get the true answers back (a maintenance record, a lab result, an inspection). Arrange that feedback loop at the start. Where it is slow, monitor the inputs instead.
Watching the inputs: the population stability index
When answers are slow to arrive, compare the distribution of an input now against its distribution in training. The population stability index (PSI) sums (now − training) × ln(now ÷ training) over bins.
Worked example: vibration RMS has shifted
| Vibration RMS bin (mm/s) | Training share | Last month | PSI contribution |
|---|---|---|---|
| < 1.5 | 18 % | 6 % | 0.1318 |
| 1.5 to 2.0 | 34 % | 20 % | 0.0743 |
| 2.0 to 2.5 | 30 % | 30 % | 0.0000 |
| 2.5 to 3.0 | 14 % | 28 % | 0.0970 |
| > 3.0 | 4 % | 16 % | 0.1664 |
- PSI = 0.470. A common rule of thumb: below 0.1, no shift; 0.1 to 0.25, a moderate shift worth a look; above 0.25 a large shift. Here the machine now spends far more time above 2.5 mm/s than in the training data.
- A big shift does not by itself mean the model is wrong (the machine may really be worse, which is what the model should detect), but it means the model is now being asked about conditions it saw rarely, so check its accuracy there.
- Also watch the share of missing or out-of-range inputs, the rate of alerts, and the average predicted probability.
Worked example: an input outside anything seen
A feature had mean 4.0 and standard deviation 0.5 in the training data. A new input of 6.2 has z = (6.2 − 4.0) ÷ 0.5 = 4.4. A simple guard: if any input is beyond ±4 to 5 standard deviations or outside its physical range, flag the prediction as unreliable instead of trusting it, and log the event.
- Log the inputs, the model version, the output and (later) the true outcome for every prediction; without them you cannot debug or retrain (see Telemetry, Databases and Dashboards).
- Track the false-alarm rate: at 1.0 % false alarms on hourly readings, 24 a day gives about 88 false alarms a year; people stop reacting long before that (see the base-rate example in Classification and Metrics).
- Set limits on the monitored numbers and alert on them, as for any other instrument.
Updating, versioning and rolling out
- Version everything: the code, the training data (or its hash), the feature code, the scaler constants and the model file, so any prediction can be traced to what produced it.
- Run in shadow mode first. The new model makes predictions that are logged but not acted on, beside the old method. After 30 days it agrees with the old decision 94 % of the time; examine the 6 % of disagreements by hand before switching.
- Roll out gradually (one machine, then a few, then all) and keep the old version ready to restore.
- Retrain on a schedule or on a trigger (drift detected, a repair done, a new product). Retrain with new data added, and re-run the full validation (see Overfitting, Validation and Tuning); a retrained model is a new model.
- Over-the-air updates of a microcontroller need signed images, a rollback, and power-fail safety (see Embedded Systems).
- Write a one-page “model card”: intended use, training data, metrics, known weaknesses, owner, and how to switch it off.
Limits and responsible use
- Not a safety function. A learned model has no guaranteed behavior and can fail on inputs unlike its training data. Where a wrong answer can hurt people, protection must come from rated, deterministic devices (see IEC 61508 and ISO 13849); keep the model in an advisory or monitoring role. See System Integration.
- Explainability. Know why it flagged something: simple models and good features make that easy; for complex ones use feature importance and example cases, and check that the model relies on sensible signals and not on a quirk of the data (see Trees, Clustering and Anomaly Detection).
- Ask what an error costs and decide in advance what action follows an alert, who checks it and what false-alarm rate people will tolerate (see What Machine Learning Is and When to Use It).
- Bias and coverage. A model is only as good as the situations in the data; check performance on each machine, shift, product and season separately, not just overall.
- Data rights and privacy. Check who owns the data and whether it includes personal information (camera images of people, operator IDs, location); minimize what you keep, secure it and state the purpose.
- Keep a human in the loop for decisions that matter, at least until the system has a long documented record, and keep the ability to override it.
- Security. A model and its data are assets; protect them, and treat inputs from outside as untrusted (adversarial inputs can fool models).
- Regulation. Some sectors and regions have rules for automated decisions and for AI systems; check the requirements that apply to your product before you deploy.
Troubleshooting
| Symptom | Likely cause | Try |
|---|---|---|
| Great test score, poor real-world results | Leakage, test data too similar to training, conditions changed | Re-split by time or machine; check for duplicates and leaked fields; test on new data (see Data Preparation, Splitting and Leakage) |
| Training error low, validation error high | Overfitting | More data, a simpler model, regularization, early stopping (see Overfitting, Validation and Tuning) |
| Both errors high | Underfitting, weak features or noisy labels | Better features (see Features for Signals and Machines), check labels, a more flexible model |
| High accuracy but misses the faults | Class imbalance | Use precision, recall and F1; reweight; set the threshold deliberately (see Classification and Metrics) |
| Model worsens over months | Drift | Monitor inputs and outputs; retrain on recent data; check the sensors |
| Results differ between PC and device | Quantization, different pre-processing, numeric differences | Run the same recorded windows through both and compare; use the same feature code |
| Too many false alarms | Threshold too low, “normal” not fully captured | Raise the threshold; include more operating modes in training; add context inputs; use persistence |
| Out of memory on the device | Activations too large or model too big | Quantize, shrink the input, reduce channels, use a bigger part; read the arena size from the tool |
| Inference too slow | Too many MACs, no optimized kernels, slow clock or memory | A smaller or depthwise network, CMSIS-NN or vendor kernels, a faster part or an accelerator |
| Cannot reproduce a result | Random seeds, changed data or library versions | Fix seeds; version data and code; record the environment |
Related
- Machine Learning: the overview of this guide.
- Neural Networks and Deep Learning: parameters, MACs and quantization for networks.
- Features for Signals and Machines: the feature code that has to run on the device.
- Embedded Systems: memory, timers, low power and over-the-air updates.
- Data Logging: logging inputs and outputs for monitoring and retraining.
FAQ
Can I train on a microcontroller?
Generally no. Train on a PC or in the cloud, then run (infer) on the microcontroller. A few simple methods, such as updating a running mean and variance for a z-score detector, can run on the device.
How do I know when to retrain?
When monitoring shows drift (a rising error against real outcomes, a large PSI on key inputs, a changed alert rate), after a repair or a change of process, and on a regular schedule as a backstop.
Will quantization ruin my accuracy?
Usually it costs under a percentage point for 8-bit post-training quantization of a well-behaved network. Measure it on the test set and on the device; use quantization-aware training if the loss is larger.
Is it safe to use a model for a protective function?
No, not on its own. Use certified safety devices for protection and keep the model for monitoring and advice. If a learned function is part of a safety system at all, it needs a formal safety case and independent safeguards.
References
- Warden, P. and Situnayake, D., TinyML, O’Reilly; the TensorFlow Lite for Microcontrollers and Edge Impulse documentation.
- Sculley, D. and others, “Hidden technical debt in machine learning systems”, NIPS, 2015; Gama, J. and others, “A survey on concept drift adaptation”, ACM Computing Surveys, 2014.
- IEC 61508 and ISO 13849-1 (functional safety); ISO 13373 series (condition monitoring and diagnostics of machines).
Last reviewed: October 2026. Times, currents and thresholds in the worked examples are typical and illustrative: measure your own system.