Storage, Formats and Data Rates

How much data?

Estimate the volume before choosing hardware or format. The data rate is channels × bytes per sample × samples per second; multiply by 86 400 for a day.

bytes per day = channels × bytes per value × samples per second × 86 400

File formats compared

Format	Size	Strengths	Weaknesses
CSV / text	Large (3 to 10 × binary)	Opens anywhere; readable; easy to repair; one line per sample	Slow to write and parse; no types; time zone and units are your job
Binary, fixed record	Smallest	Fast, constant row size so you can seek by index; ideal for microcontrollers	Needs a documented layout and the right tools to read; endianness
HDF5 / NetCDF	Compact, compressed	Self-describing, metadata and many channels in one file, chunked access	Library needed; not for small microcontrollers; a crash can corrupt the file
Parquet / Arrow	Compact, columnar	Fast analysis with pandas and Spark; typed; compressed	Written in blocks, so unsuitable for streaming a crash-safe log
JSON lines (NDJSON)	Large	One record per line; flexible fields; easy for web tools	Larger and slower than CSV for numbers
Time-series database	Compact on disk	Queries, downsampling, retention and dashboards built in	A service to run, update and back up
SQLite	Moderate	One file with SQL; transactions; good for small loggers	Write rate and flash wear need care
TDMS, WAV, vendor formats	Compact	Used by instruments and tools you may already own	Tied to one ecosystem

File parsing code (Python, C, pandas) is in Data, Files and Parsing. For a first design: CSV for slow data, fixed-record binary for fast data, a database for fleets of loggers.

Worked example: sizes

// 20-byte fixed record on a microcontroller (little-endian, packed)
struct __attribute__((packed)) Record {
    uint32_t t;          // seconds or ticks since the file start
    uint16_t ch[8];      // raw ADC counts
};
static_assert(sizeof(Record) == 20, "record size");
// Write whole records only; add a small header (version, scale, start time) at the start of each file.

Compression

Slowly changing data compresses very well because neighboring values are similar. Delta coding (store the difference from the previous value) followed by a general compressor often shrinks a trend log by 4 to 10 times; random-looking noise hardly compresses.

• Round to the real resolution: storing 23.4561234 °C from a ±0.5 °C sensor wastes space and suggests accuracy that is not there.
• Store changes, not levels, when the signal is steady (see Logging Strategies: Triggers, Deadbands and Aggregation).
• Columnar formats (Parquet) compress each channel on its own and work well for analysis.

Storage media and endurance

Medium	Strengths	Cautions
SD / microSD card	Cheap, large, removable, simple	Wear-limited; corrupts if power fails during a write; quality varies. Use industrial or high-endurance cards for continuous logging
eMMC / SSD	Fast, reliable, larger endurance	Needs a proper host; SSDs can lose data on sudden power loss unless they have power-loss protection
Serial flash / FRAM (on board)	Tiny, cheap; FRAM has near unlimited writes	Small capacity; flash must be erased in blocks
USB stick	Easy to collect	Poor endurance, fragile connector, inconsistent quality
Network storage / cloud	Large, backed up, remote	Needs a connection and buffering (see Telemetry, Databases and Dashboards)
Hard disk	Cheap per terabyte	Moving parts; shock and vibration; slow to start

File handling: rotation, flushing and recovery

• Rotate files (a new file per hour or per day, named by UTC time, cr1_2026-10-31T10.csv): a damaged file loses only a short span, and old files can be moved, compressed and deleted without touching the live one.
• Flush on a schedule, not every sample and not only at the end: flush() + fsync() (or f_sync() on FatFs) every few seconds trades a small loss on power failure for far less wear and delay.
• Write a header in every file: logger ID, firmware, start time, channel names, units and scaling, so that each file is complete without the others.
• Append-only, never rewrite earlier data. If you must update an index, write it to a second file.
• Include a counter or checksum per record or per block, so that damage is detected (see Data Quality, Analysis and Reporting and the CRC examples in Embedded Systems).
• Check free space and stop or rotate gracefully rather than failing silently when the card is full; log the event.
• On restart, start a new file; do not append to a file whose last record may be incomplete.

Databases

A database handles many loggers, many users, queries and retention. A time-series database (InfluxDB, TimescaleDB, Prometheus) stores each measurement against its time and compresses well; a relational database (PostgreSQL, MySQL) suits events, metadata and reports. Row-by-row inserts are slow: batch the writes.

Retention and backup

• Back up off-site and test a restore; a backup that has never been restored is a hope.
• Keep the original raw files for anything legal, regulatory or contractual; summaries are for convenience.
• Plan for the data to outlive the logger: document the format, keep a reader program, and prefer open formats (CSV, HDF5, Parquet).

Related

• Data Logging: the overview of this guide.
• Logging Strategies: Triggers, Deadbands and Aggregation: storing less without losing the information.
• Logger Hardware: Dedicated, Embedded and Industrial: the logger platforms and their storage.
• Data, Files and Parsing: reading and writing these formats in code.
• Embedded Systems: writing files and CRCs on a microcontroller.

FAQ

CSV or binary?

CSV for slow data and anything people will open in a spreadsheet; fixed-record binary for fast data on microcontrollers; a database or HDF5/Parquet for large, queryable archives. Write a header with units and scaling whichever you choose.

How big a card do I need?

Channels × bytes × rate × 86 400 × the days between collections, doubled for margin. A card should stay under about 70 % full.

Why did my SD card corrupt?

Most often because power was lost during a write. Flush on a schedule, write whole sectors, use a supercapacitor to finish writing, and start a new file at every restart.

How long should I keep data?

As long as it may be needed: keep raw data for weeks to months, summaries for years, and anything with a legal or contractual requirement for the required period.

References

• SD Association, Physical Layer Simplified Specification; manufacturers’ industrial SD endurance notes.
• The HDF Group, HDF5 documentation; Apache Parquet documentation; InfluxData and Timescale documentation on retention and downsampling.
• Kleppmann, M., Designing Data-Intensive Applications (storage and durability).

Last reviewed: October 2026. Card endurance and database sizes are typical and vary widely between products: check the datasheet and measure.