Telemetry, Databases and Dashboards
MQTT, store-and-forward, time-series databases, Grafana dashboards, alerts, data costs and securing a remote logger.
The path from logger to screen
Getting data from a remote logger to a person involves five stages: a transport (Wi-Fi, cellular, LoRa, Ethernet), a protocol (MQTT, HTTP), a broker or gateway, a database and a dashboard or alert. Each stage can fail, and the design goal is that a failure anywhere delays the data but never loses it.
| Link | Typical range | Data rate | Power | Notes |
|---|---|---|---|---|
| Wi-Fi | 30 m indoors | Mbit/s | Medium | Needs a network and a password; fine for buildings |
| Ethernet | 100 m per run | 100 Mbit/s | Low (PoE) | The most reliable; PoE gives power and data in one cable |
| Cellular (LTE-M, NB-IoT, 4G) | Anywhere with coverage | kbit/s to Mbit/s | High when transmitting | Per-megabyte cost; coverage and certification |
| LoRa / LoRaWAN | Kilometers | 0.3 to 50 kbit/s | Very low | Tiny messages and duty-cycle limits; a gateway |
| Bluetooth Low Energy | 10 m | Up to 1 Mbit/s | Very low | Phone or gateway reads the logger |
| RS-485 / Modbus | 1 km | Up to 115 kbit/s | Low | Industrial wired networks (see Industrial Communication) |
| Satellite | Global | Bytes per message | High | Costly per message; for very remote sites |
Protocols: MQTT, HTTP and others
| Protocol | Style | Strengths | Cautions |
|---|---|---|---|
| MQTT | Publish / subscribe through a broker | Light, keeps a connection, handles intermittent links, quality-of-service levels, retained and last-will messages | Needs a broker (Mosquitto, EMQX, a cloud service); secure it |
| HTTP / HTTPS (REST) | Request / response | Simple, works through firewalls, easy to test with curl | Overhead for each reading; the logger must retry |
| CoAP | Light request / response over UDP | Tiny messages for constrained devices | Less tooling; UDP and NAT issues |
| OPC UA | Client / server or pub-sub, with information models | The industrial standard; typed and secure | Heavier; license-free stacks exist but need effort |
| Modbus TCP | Poll registers | Ubiquitous in plants | Polling only; no security of its own |
| File upload (SFTP, rsync, S3) | Send whole files | Robust for big files, trivially resumable | Delay of a file period; not real time |
# Topic and payload convention (MQTT), then the same shape in a database tag set
topic : site/coldroom1/logger3/telemetry
payload : {"id":"CR3","t":"2026-10-31T10:40:00Z","air":4.21,"prod":3.87,"hum":71.2,"door":0,"v":3.91,"rssi":-84}
# Quality of service: 0 = at most once, 1 = at least once (duplicates possible), 2 = exactly once (slowest).
# Use QoS 1 and make the receiver idempotent (the same time + id twice stores one row).- Name topics in a hierarchy (
site/area/device/measurement) so that subscriptions and permissions can use wildcards. - Send the logger’s own time stamp and the measurement identifiers with each message (see Time Stamps and Clocks); never rely on arrival time.
- Use a last-will message and a periodic status message, so that a dead logger is noticed by the broker rather than by silence.
- Retained messages keep the last value for a new subscriber; useful for status, not for history.
Reliability: store and forward
Networks fail. The logger must write every record to local storage first and mark it as sent only when the receiver has acknowledged it. This pattern, store and forward, turns an outage into a delay.
Worked example: a six-hour outage
- A logger sends one 102-byte record a minute. A 6-hour outage leaves a backlog of 360 records (36.7 kB).
- On reconnecting, upload the backlog and keep up with new records. At 20 records per second, catching up takes about 18 s. Throttle the catch-up so that live data and the radio are not swamped.
- Retry with exponential backoff: wait 5, 10, 20, 40, 80, 160 … s (capped at 300 s) between attempts, with random jitter, so that a hundred loggers do not all hammer the server the moment it returns.
- Make the receiver idempotent: a retry after a lost acknowledgment sends the same record twice. Use the logger ID + time stamp (or a sequence number) as the key, and discard duplicates.
- Size the local buffer for the longest outage you will accept (a week of data?) and decide what to drop first when it is full (oldest raw data, never alarms).
- Watch the queue length: report it as a status value; a growing backlog is an early warning of a network fault.
- Keep a monotonic sequence number, so that missing records can be spotted at the server (see Data Quality, Analysis and Reporting).
Cost of the data
On cellular or satellite links the data volume is a running cost, and protocol overhead can exceed the payload. A TLS connection per reading is expensive; keep one open, or batch.
Worked example: a month of cellular data
- One JSON record is
102bytes; with about 60 bytes of MQTT and TLS overhead per publish it is 162 bytes. Once a minute for 30 days: 7.0 MB. - Batching 10 records per message pays the overhead once: 1.1 kB per message, 4.7 MB a month.
- Binary payload (a 4-byte time and six 2-byte values = 16 bytes) cuts the record further: 3.3 MB a month unbatched.
- The cheaper plan is the one with the right number of megabytes, and a pooled plan across many loggers. Check that the logger does not retry in a tight loop when the signal is poor; that is how a SIM card uses a month’s allowance in a day.
Time-series databases
| Database | Query language | Good for | Notes |
|---|---|---|---|
| InfluxDB | Flux / InfluxQL / SQL | Sensor streams with tags, retention policies, downsampling | Free and cloud versions; tags are indexed, fields are not |
| TimescaleDB (PostgreSQL) | SQL | Time series next to ordinary relational data | Full SQL and joins; hypertables partition by time |
| Prometheus | PromQL | Metrics and alerts for servers and devices | Pull-based; limited long-term storage on its own |
| SQLite / PostgreSQL / MySQL | SQL | Small systems and event tables | Index on time; batch the inserts (see Storage, Formats and Data Rates) |
| Cloud IoT services (AWS, Azure, Google, ThingSpeak, Adafruit IO) | Service specific | Fast start; storage and charts included | Per-message cost and lock-in; look at the export |
| Industrial historians (PI, Ignition, Wonderware) | Vendor | Plant-wide history with compression | Licensed; the standard in process industry |
- Separate the series from its metadata: tags such as site, device and channel are searchable; units, calibration and location live in a table (see Planning a Logging System).
- Plan retention and downsampling at the start: raw for days, minute aggregates for months, hourly for years (see Storage, Formats and Data Rates).
- Avoid high cardinality in tags (a unique ID per message is not a tag); it makes the index huge.
- Back up the database and test the restore; do not rely on the cloud to be the only copy.
Dashboards
Grafana, Node-RED dashboards, Home Assistant and the services’ own pages turn a database into charts and gauges. A dashboard is for answering a question quickly, so choose what to show, not everything you can.
Worked example: a year on one chart
One minute data for a year is 525 600 points per series. A chart 1 200 pixels wide can show only about 1 200: each pixel hides 438 samples (about 7.3 hours). Ask the database for the mean, min and max per bucket (a downsampled query) instead of sending 525 600 points to the browser. Showing only the mean would hide excursions (see Logging Strategies: Triggers, Deadbands and Aggregation).
- Lead with the answer: the current value, the state (OK, warning, alarm), and one trend chart for the last day; details a click away.
- Show units, the time range and the age of the data (“updated 2 min ago”), and turn a stale value gray; a number that stopped updating looks normal.
- Mark thresholds and set-points on the chart; use the same color for the same channel on every panel.
- Show gaps as gaps, not as a straight line across them (see Data Quality, Analysis and Reporting).
- Keep it readable on a phone and a wall screen; use large numbers for the few that matter.
Alerts that people trust
Worked example: an alert rule
- Alert if the cold-room air temperature is above 5 °C for more than 15 minutes: with a 60 s sample that is 15 consecutive readings, which ignores a door opening.
- Use hysteresis (alarm above 5, clear below 4.5), or a value hovering at the limit sends an alert every minute.
- Alert on missing data too: “no data for 3 × the interval” is the most important rule, because a dead logger gives no readings to alarm on.
- Send the alert to a person who can act, with the value, the limit, the time and a link to the chart; escalate if nobody acknowledges.
- Too many alerts teach people to ignore them. Review each alert: if nobody acts on it, remove it or raise its threshold.
- Test the chain (trigger a real alert, check the email or text arrives) and repeat after any change.
- An alert is not a safety system (see the safety notes in Data Logging).
Securing a remote logger
| Risk | Defense |
|---|---|
| Data sent in the clear | TLS on MQTT (port 8883) and HTTPS; check the server certificate |
| Weak or shared credentials | A unique username and key per device; never put secrets in the firmware you publish; rotate them |
| Open ports on the logger | Make outbound connections only; no inbound port forwarding; or use a VPN |
| An open broker or database on the internet | Authentication and access control lists (a device may publish only to its own topic); a firewall |
| Old software | Updates and signed firmware (over-the-air updates with rollback, see Embedded Systems) |
| A logger on a control network | Read-only access, a separate network segment, a DMZ; agree it with the controls and IT staff |
| Personal or confidential data | Collect only what you need, restrict who can see it, follow the rules for energy, location and occupancy data |
Related
- Data Logging: the overview of this guide.
- Storage, Formats and Data Rates: formats, retention and databases.
- Logger Hardware: Dedicated, Embedded and Industrial: choosing the logger and its link.
- Industrial Communication: OPC UA, MQTT and network security in plants.
- Embedded Systems: Wi-Fi, MQTT and over-the-air updates on a microcontroller.
FAQ
MQTT or HTTP?
MQTT for continuous telemetry from many devices over unreliable links (it holds a connection and has delivery levels). HTTP for occasional uploads and where firewalls or simplicity matter.
How do I avoid losing data when the network drops?
Write every record to local storage first, send in the background, remove a record only after the server has acknowledged it, and make the server ignore duplicates.
Do I need the cloud?
No. A local database and dashboard on a Raspberry Pi or a server works well and costs little. The cloud helps with access from anywhere, backups and scale; compare the running cost.
Why is my dashboard slow?
It is probably asking for every raw point. Query downsampled buckets (min, mean, max) that match the chart width, and add an index on time.
References
- OASIS, MQTT Version 5.0 specification; the Eclipse Mosquitto and HiveMQ documentation.
- InfluxData, TimescaleDB and Grafana Labs documentation; RFC 8446 (TLS 1.3).
- IEC 62443 (industrial cybersecurity); NIST SP 800-82, Guide to Operational Technology Security.
Last reviewed: October 2026. Prices, overheads and message sizes are typical; measure your own link and check the provider’s terms.