Debugging, Testing and Troubleshooting

A debugging method

• 1. Reproduce it. A bug you can trigger on demand is half fixed. Note exactly what you did, the inputs, the supply and the temperature. If it is intermittent, look for what changes: timing, supply, a particular input.
• 2. Split the problem in two and find which half holds the bug: hardware or firmware? This module or the next? Binary search through the system is far faster than reading code.
• 3. Look before you change. Measure the supply, the clock and the signals; read the values; do not guess. Most “firmware bugs” on a new board are wiring, power or a misconfigured pin.
• 4. Change one thing at a time and write down the result. If you changed three things and it works, you do not know which one fixed it.
• 5. Check your assumptions: the datasheet (pin, address, mode, byte order, units), the units of a variable, the size of a type.
• 6. Fix the cause, not the symptom, then add a test or an assertion so it cannot come back unnoticed.

The tools and what each shows

Tool	What it shows	When to use it
Serial print / log	Text messages from the program	First look at values and flow; remember that it takes time and can change timing
LED or pin toggle	That a point in the code was reached, and when (on a scope)	Very early start-up, interrupt timing, no serial port available
Multimeter	DC voltage, continuity, current	Check supply rails, connections and pin levels
Logic analyzer	Digital signals over time, with decoding for UART, I²C, SPI and CAN	Bus problems, timing, wrong addresses, modes and framing
Oscilloscope	Voltage waveforms: edges, noise, ringing, supply droop	Signal quality, PWM, analog noise, reset causes, power glitches
Hardware debugger (SWD or JTAG)	Breakpoints, registers, memory, step through code	Crashes, hard faults, and anything that print statements cannot explain
Current profiler	Current against time	Power problems and unexpected wake-ups (see Power and Low-Power Firmware)
Protocol sniffer	Bluetooth, Wi-Fi, Modbus and CAN traffic	Wireless and fieldbus faults (see Industrial Communication)

Print debugging and logging

Printing a value is the oldest debugging tool and still the first one to reach for. Its cost is time: serial output is slow and often blocks.

• Log levels: error, warning, info and debug, switchable at build time, so the noisy ones cost nothing in production.
• Time stamp each message with a running counter so you can see the order and gaps.
• Print the unexpected values, not only the expected: a variable’s value on entering an error branch is the best clue.
• Do not print from an ISR, and do not trust a bug that only appears with the print removed: it is timing.

// A RAM ring buffer of events with time stamps: cheap enough to leave in production
struct Event { uint32_t t_ms; uint8_t code; uint8_t pad; uint16_t arg; };   // 8 bytes
#define LOG_N 64
static struct Event logbuf[LOG_N];
static uint8_t log_i = 0;

static inline void log_event(uint8_t code, uint16_t arg) {
  logbuf[log_i] = (struct Event){ millis(), code, 0, arg };
  log_i = (log_i + 1) & (LOG_N - 1);            // LOG_N is a power of two
}
// On a fault or on request, dump the last LOG_N events in order: 512 bytes of RAM for the last 64 events

Logic analyzers and oscilloscopes

A logic analyzer records digital signals with a timestamp and decodes the protocols; an oscilloscope shows the analog voltage, so you see ringing, slow edges, noise and droop. Use the analyzer to learn what was sent and the scope to learn whether the signal is clean.

• Trigger on what you are hunting: a falling edge on the chip select, a particular byte, a glitch shorter than a given width.
• Use the decoder: I²C, SPI, UART, CAN and 1-Wire analyzers decode the bytes, flag NAKs and framing errors and export the data.
• Check the supply while a load switches: a dip of a few hundred millivolts at the moment the radio or a motor turns on explains many mysterious resets.
• Soft analyzer: a cheap 8-channel 24 MS/s USB analyzer and free software (PulseView, Saleae Logic) pays for itself the first time a bus misbehaves.

Debug probes: SWD, JTAG and GDB

A debug probe (ST-Link, J-Link, CMSIS-DAP, a Raspberry Pi Picoprobe, ESP-Prog) connects to the chip’s two-wire SWD or four-wire JTAG port. With it you can program the chip, halt the processor, step through source code, set breakpoints (stop at a line) and watchpoints (stop when a variable changes), read and write memory and registers, and inspect the call stack. A debugger talks to the probe through GDB and OpenOCD, or through an IDE (VS Code with Cortex-Debug, PlatformIO, STM32CubeIDE, Keil, IAR).

• Hardware breakpoints are few (typically 4 to 8 on a Cortex-M), because they are comparators in the chip; flash breakpoints beyond that are slow.
• Optimization hides variables: build with -Og or -O0 for debugging; “optimized out” means the compiler removed it.
• Stopping changes the system: halting a motor controller leaves the outputs where they were, which can be dangerous; set the outputs safe first, or use a hardware watchdog setting that freezes during debug.
• Trace (ITM/SWO, ETM) records what ran without stopping, which is the way to find a timing bug.
• Read the registers of a peripheral in the debugger (SVD files give names) to see what the hardware thinks it is doing.

Hard faults and crashes

When the processor does something illegal (reading a bad address, executing garbage, an unaligned access, a divide by zero), it takes a fault exception. On an ARM Cortex-M the hard-fault handler is the place to catch it: it receives the saved registers, including the program counter (PC) at the time of the fault, which points at the guilty instruction.

// Cortex-M hard-fault handler that records the stacked registers, then resets
void HardFault_Handler(void) {
  __asm volatile(
    "tst lr, #4        \n"       // which stack was in use?
    "ite eq            \n"
    "mrseq r0, msp     \n"
    "mrsne r0, psp     \n"
    "b hard_fault_c    \n");
}
void hard_fault_c(uint32_t *stk) {            // stk[6] = PC, stk[5] = LR, stk[0..3] = R0..R3
  saved.pc = stk[6]; saved.lr = stk[5];
  saved.cfsr = SCB->CFSR;                     // why: precise bus fault, undefined instruction ...
  saved.bfar = SCB->BFAR;                     // the bad address
  NVIC_SystemReset();                         // after writing the log to retained RAM
}

• Save the evidence: store the PC, LR and fault status in retained RAM or flash, and report them on the next start.
• Enable the fault handlers (usage, bus, memory-management faults) so that you get a precise cause instead of a generic hard fault.
• Stack overflow faults are best caught with the memory-protection unit or a stack guard; see the stack canary in Memory, Number Formats and Reliability.

Testing firmware

Firmware can be tested like any other software, if you keep the logic apart from the hardware (see Platforms and Toolchains). Compile the decision code on your PC with a unit-test framework (Unity, CppUTest, GoogleTest), feed it recorded or simulated inputs and check the outputs: thousands of cases run in a second, with a real debugger and no hardware.

// Unity test on the PC: bouncing press must not register until 8 stable samples
void test_debounce_press(void) {
  int levels[] = {1,1,0,1,0,0,0,0,0,0,0,0,0,0};
  reset_debouncer();
  for (int i = 0; i < 14; i++) {
    fake_pin = levels[i];
    debounce_tick();
    if (i < 11) TEST_ASSERT_FALSE(pressed);   // the 8 stable zeros are samples 4 to 11
  }
  TEST_ASSERT_TRUE(pressed);
}

• Static analysis (cppcheck, clang-tidy, compiler warnings with -Wall -Wextra) finds uninitialized variables, overflows and unused code without running anything.
• Host sanitizers (AddressSanitizer and UndefinedBehaviorSanitizer) run the portable code on the PC and report memory errors and undefined behavior at once.
• Coverage shows which code your tests never ran: the bugs live there.
• Keep a regression test for every bug you fix.
• Simulators and emulators (QEMU, Renode, Wokwi) run firmware without a board for early development and automated tests.

Debugging devices in the field

• Record why it reset (power-on, brown-out, watchdog, software) and count the resets; report them to a server or show them on a status screen.
• Keep a log of the last events in retained RAM or flash (the ring buffer above) and send it with the fault report.
• Expose counters for errors that should be rare (CRC failures, bus timeouts, retries, free stack and heap): a rising counter warns before a failure.
• Version everything: firmware version, build date and git hash are printed at start-up and sent with every report.
• Remote update (see Wireless, Networking and Updates) so that a fix reaches the device without a visit.
• Safe defaults and a recovery mode: a button or a command that returns the device to a known state.

Symptoms, causes and fixes

Symptom	Likely cause	Try
Nothing happens after flashing	Wrong board or clock selected, pin mapping wrong, a boot-mode pin held, program crashed at start-up	Blink an LED from the first line of main; check the crystal and the reset pin; check the boot pins; use a debugger to see where it stops
Random resets	Brown-out, missing decoupling, watchdog not kicked, stack overflow, motor noise	Watch the supply on a scope while loads switch; add capacitors; check the watchdog and stack use; read the reset-cause register
Input reads random values	Floating pin, no pull-up or pull-down, long wire	Enable the pull-up or add a resistor; shorten the wire; add a filter (see GPIO, Interrupts and Debouncing)
Garbage on the serial port	Baud rate mismatch, wrong voltage level, TX and RX not crossed	Check the baud rate and 8N1 framing; cross TX to RX; check levels (see UART, I²C, SPI and CAN in Firmware)
I²C device not found	Wrong address, missing pull-ups, swapped SDA and SCL, voltage mismatch	Run an I²C scanner; check pull-ups with a scope; check 3.3 V versus 5 V
SPI returns all 0s or all 1s	Wrong mode, CS not asserted, wrong wiring, slave not powered	Check the mode and CS on a logic analyzer; check MISO is connected and the device powered
ADC readings noisy or drifting	High source impedance, switching noise, shared supply, wrong reference	Add a capacitor, average, keep analog wiring away from motors, use a stable reference (see Analog Input and Signal Processing)
PWM is not the expected frequency	Prescaler or timer mode different from assumption, library default	Read the timer registers; measure with a scope (see Timers, PWM and Timing)
Works on USB, fails on battery	Voltage too low or too noisy, regulator dropout, current spikes	Measure the supply under load; use a larger capacitor or a better regulator
Works for hours or days, then hangs	Memory leak or fragmentation, stack overflow, counter overflow, a lock never released, a race	Log free memory and stack over time; avoid dynamic allocation; add a watchdog and a log; check the 49-day millis() roll-over
Works with the debugger, fails without	Timing change from halting, a pin used by the debug port, a watchdog frozen by the debugger, uninitialized memory	Disable the debugger’s side effects; check what the debugger sets; use trace instead of stopping
Works in debug build, fails in release	Optimization exposes a missing volatile, an uninitialized variable or undefined behavior	Compile with warnings and sanitizers; check volatile on shared variables; try -O2 in debug
Wi-Fi or BLE drops out	Weak signal, power sag during transmit, blocking code starving the network task	Check the supply with a scope during transmit; keep tasks short; log the disconnection reason

Related

• Embedded Systems: the overview of this guide.
• Firmware Architecture, State Machines and RTOS: watchdogs, logging and fail-safe design.
• Memory, Number Formats and Reliability: stack overflow, CRCs and settings.
• Power and Low-Power Firmware: measuring current and finding unexpected wake-ups.
• Test Tools and Fault Finding and Troubleshooting Flowcharts: meters, scopes and the hardware side of finding faults.

FAQ

Why does my program work with a print statement and fail without it?

The print changes the timing (it takes milliseconds at a typical baud rate), hides a race, or changes the compiler’s optimization. Suspect a missing volatile, a race with an interrupt, or an uninitialized variable, and use a pin toggle or a RAM log instead.

Do I need a logic analyzer?

For anything with a serial bus, yes: a cheap 8-channel USB analyzer with protocol decoding turns hours of guessing into a minute of looking. An oscilloscope is the next tool, for signal quality and supply problems.

How do I find a crash that happens once a day?

Make the evidence survive: enable the fault handlers, store the PC and fault status in retained memory, keep a ring buffer of recent events, record the reset cause, and read them after the next failure. A watchdog turns a hang into a reset with a log.

Can I unit test code that touches hardware?

Test the logic on a PC by hiding the hardware behind small functions (read pin, write pin, get time) that tests replace with fakes. Then test the drivers on the real board with a loop-back or a second board.

References

• White, E., Making Embedded Systems, O’Reilly (debugging, testing and tools); Grenning, J., Test Driven Development for Embedded C, Pragmatic Bookshelf.
• Arm, Cortex-M Generic User Guide and Application Note on hard-fault analysis; SEGGER RTT and Cortex-Debug documentation.
• Ganssle, J., The Art of Designing Embedded Systems; Saleae and Sigrok documentation (logic analyzer protocol decoding); Tektronix, XYZs of Oscilloscopes.

Last reviewed: October 2026. Costs and timings use typical values: measure on your own hardware.