Production firmware should be designed to fail in a way that leaves evidence. A useful fault architecture captures enough processor and system state to reconstruct the failure, protects that evidence across reset, restores service through a controlled policy, and preserves the exact build information needed to symbolicate the crash later.

This lesson follows OSFEC.009: CAN Bus Engineering and complements OSFTC.010: Firmware Crash Logs and Fault Recovery. The technician lesson focused on preserving crash evidence; this engineering lesson focuses on designing firmware so that evidence exists, survives reset, can be decoded automatically, and does not create a second reliability or security problem.

Memfault demonstrates practical Cortex-M HardFault analysis, stacked-register recovery, GDB use, memory-corruption debugging, and crash collection at scale.

Elementary Overview

A firmware crash handler is like a flight recorder. It should save the important facts about what the processor was doing before the system resets. The engineer decides which facts to keep, where to keep them, how to prove they belong to the correct firmware build, and when the device is allowed to restart.

Learning Objectives

• Design a Cortex-M fault handler that preserves exception state without corrupting additional evidence.
• Explain the difference between fault capture, retained crash records, and full core dumps.
• Choose what state belongs in retained RAM, flash, or external telemetry.
• Build a symbolication pipeline tied to exact firmware build identity.
• Design watchdog and reset policy that distinguishes recovery from root-cause analysis.
• Prevent crash logging from causing flash wear, reset loops, or unsafe behavior.
• Protect sensitive information contained in production dumps.
• Validate the complete crash pipeline with deliberate fault injection.

Fault Handling Has Two Jobs

A production fault path must do two different things: preserve diagnostic state and move the system toward a defined safe recovery state. Those goals can conflict. Writing a large log may preserve more context but increase the chance of blocking inside a damaged driver. Resetting immediately may restore service but destroy volatile evidence. The architecture must decide in advance which state can be captured safely before reset and which analysis can wait until the next boot.

The first rule is to keep the fatal exception path small. It should avoid dynamic allocation, complex logging stacks, blocking buses, filesystem operations, or dependencies on subsystems that may already be corrupted. Capture raw state into a bounded structure first. Formatting, symbolication, upload, compression, and human-readable decoding can happen later after a clean reboot.

SEGGER demonstrates Cortex-M fault analysis, including nested exceptions, multiple stacks, and trace-assisted diagnosis.

Capture the Exception Frame Correctly

On Cortex-M exception entry, hardware normally stacks R0-R3, R12, LR, PC, and xPSR on the active stack. The fault handler must determine whether the exception used the Main Stack Pointer or Process Stack Pointer. On processors with floating-point support, an extended frame may also exist. Incorrectly assuming one stack layout can produce a convincing but false crash record.

The architecture should preserve the raw stacked frame plus SCB fault registers such as CFSR and HFSR, and MMFAR/BFAR when their valid bits indicate a meaningful address. Keep raw hexadecimal values even if firmware also stores a decoded reason code. Human-readable decoding can change as tooling improves; the original register state cannot be reconstructed if it was never saved.

struct crash_record {
    uint32_t magic;
    uint32_t version;
    uint32_t build_id;
    uint32_t reset_reason;
    uint32_t exc_return;
    uint32_t r0, r1, r2, r3, r12;
    uint32_t lr, pc, xpsr;
    uint32_t msp, psp;
    uint32_t cfsr, hfsr, mmfar, bfar;
    uint32_t active_irq;
    uint32_t uptime_ms;
    uint32_t crc32;
};

This is an illustrative structure, not a universal ABI. A production format should have a version, length, integrity check, explicit endianness, build identity, and compatibility rules so old crash records remain decodable after the firmware evolves.

Retained RAM Is Fast but Temporary

A small crash record can often be written to a .noinit region, backup SRAM, retention RAM, or another memory area that survives a software reset. This keeps the fatal path short and avoids flash writes while the processor may be unstable. On the next boot, startup code validates the record, copies it to durable storage or telemetry, then clears or advances the retained slot.

Retained RAM is not guaranteed across every reset. Power loss, brownout, battery-domain failure, bootloader behavior, or aggressive startup initialization may erase it. The design must know which reset classes preserve the chosen memory and must treat retained data as untrusted until its magic, version, bounds, and integrity check have been validated.

Flash Gives Persistence but Has Costs

Flash-backed crash storage survives power removal but introduces write latency, erase granularity, wear, and power-fail atomicity concerns. A device caught in a rapid crash-reset loop can consume erase cycles or overwrite the first useful crash with hundreds of identical later events. Production designs commonly use a bounded ring, pre-erased slots, monotonically increasing sequence numbers, CRCs, and write-rate limits.

A robust format should distinguish capture complete from capture interrupted. One approach is to write the payload and integrity value first, then write a small commit marker last. On boot, records lacking a valid commit marker are ignored. The exact mechanism depends on the flash technology and its atomic write size.

Core Dumps Expand the Evidence Set

A compact fault record preserves the minimum. A core dump preserves enough registers and memory to reconstruct much more of the failed execution context offline. Zephyr’s core-dump subsystem can capture CPU registers plus selected memory and can store the result through logging, UDP, flash, or other supported backends. Engineers can choose minimal thread-state capture or broader RAM coverage depending on storage, privacy, and diagnosis needs.

Memfault explains embedded coredumps, Zephyr capture, GDB analysis, and why structured post-mortem state can outperform ordinary logs for difficult field failures.

A dump does not need to contain all RAM. Engineers should deliberately choose the smallest useful set: active task stack, selected RTOS control blocks, fault registers, key queues, allocator state, or other structures relevant to the product. Smaller dumps reduce storage, bandwidth, upload time, and exposure of sensitive memory.

Build Identity Is Part of the Crash Record

Symbolication is only correct when crash addresses are resolved against the exact executable image that produced them. Every production build should have a stable identifier tied to the ELF, map file, compiler/linker configuration, source revision, and binary image. The identifier might be a Git commit plus build number, a cryptographic image digest, a generated build UUID, or another immutable value.

The backend should use that identifier to select the matching symbols automatically. Engineers should never choose an ELF manually because its filename “looks right.” Different optimization, link order, feature flags, or libraries can move addresses enough to make an incorrect symbol file produce believable but wrong source locations.

Separate On-Device Capture From Off-Device Symbolication

Production firmware usually should not carry full debug symbols. Instead, the device records raw addresses and build identity; a secure backend stores the matching ELF and performs symbolication. That separation reduces image size and prevents field devices from needing symbol databases while still supporting detailed post-mortem analysis.

arm-none-eabi-addr2line \
  -e build/firmware.elf \
  -f -C \
  0x08001234 0x08004567

arm-none-eabi-gdb build/firmware.elf
(gdb) target remote :3333
(gdb) info registers
(gdb) bt

The same principle applies to automated systems: preserve machine-readable addresses and select symbols by immutable build ID before generating human-readable traces.

The Watchdog Is Part of the Fault Architecture

A watchdog should detect loss of forward progress, not merely prove that one periodic interrupt still runs. In an RTOS system, a supervisor can require health signals from critical tasks and feed the hardware watchdog only when the system as a whole is healthy. If the watchdog fires, retained state should distinguish which health check stopped progressing and which code path last reported success.

Do not automatically extend a watchdog timeout because a workload occasionally trips it. First determine whether the timeout budget is wrong, the workload has legitimate worst-case latency, interrupts are blocked too long, a mutex deadlocks, storage stalls, or memory corruption has stopped the service path. Recovery policy must be designed from measured worst-case timing, not from a desire to make resets disappear.

Prevent Reset Loops

A device that crashes immediately after boot can enter a reset loop that prevents diagnostics, network upload, or safe operation. Engineers should define a boot-failure counter and a fallback policy. Examples include entering a recovery image, holding in a diagnostic mode, suppressing the failing feature, rolling back an update, or waiting for service intervention after a defined number of consecutive failures.

The counter itself must be designed carefully. It may live in backup registers, FRAM, flash, or another persistent mechanism. It should distinguish planned resets from crash resets and should decay or clear after a sufficiently long healthy runtime so a single historical failure does not permanently bias future recovery.

Crash Data Is Security-Sensitive

Memory dumps can contain passwords, cryptographic keys, authentication tokens, personally identifiable data, network credentials, proprietary algorithms, or customer payloads. A debugging architecture must therefore define what memory is allowed into a dump, how records are encrypted or authenticated, who can retrieve them, how long they are retained, and how they are deleted.

Do not assume that “internal debug data” is harmless. Core dumps should follow the same security and privacy review applied to logs and telemetry. In high-assurance systems, sensitive buffers may be excluded from dumps or scrubbed before persistence when that can be done safely without damaging the essential failure record.

Crash Telemetry Needs Backpressure

Fleet telemetry can become a denial-of-service problem if one firmware bug causes millions of identical crash uploads. Backends should deduplicate crashes by build ID and signature, rate-limit repeated events, preserve occurrence counts, and retain representative dumps rather than uploading every copy. Devices should also bound local retry queues so a network outage does not fill storage with stale diagnostics.

A useful crash signature may combine build ID, exception class, normalized PC, a few frames of the call stack, and selected status bits. The signature should remain stable for repeated instances of the same failure while avoiding accidental grouping of unrelated faults.

How do you actually debug HardFault dumps day to day? (manually decoding CFSR/HFSR, or do you have better tools?)
byu/ToughProfessional544 inembedded

An embedded-systems discussion illustrates common field practice: save the stacked PC first, retain crash state across reset, automate fault-register decoding, and use matching ELF symbols instead of relying on a generic HardFault message.

Test the Failure Path Deliberately

A crash system that has never been intentionally triggered is not verified. Build a fault-injection test matrix for development and CI hardware. Inject known null dereferences, invalid memory reads, divide-by-zero where supported, stack overflow, assertion failure, watchdog timeout, nested-fault conditions, and power interruption during crash-record persistence. Every test should have an expected reset cause, expected record fields, and expected post-boot behavior.

The Linux Foundation and Memfault show Zephyr core-dump configuration, memory selection, storage backends, and GDB-based post-mortem analysis.

Engineering Validation Matrix

• Fault-frame correctness: PC, LR, stack selection, xPSR, and status registers match debugger truth.
• Persistence: retained records survive every intended reset class and are rejected after unintended power loss when incomplete.
• Integrity: corrupted records fail CRC/signature validation cleanly.
• Build mapping: each record selects one exact ELF and cannot silently fall back to a nearby build.
• Wear control: repeated crash loops cannot exhaust flash endurance quickly.
• Recovery: boot-loop policy reaches the designed safe or rollback state.
• Security: dump collection excludes or protects sensitive regions according to policy.
• Telemetry: duplicate crashes are counted and deduplicated rather than flooding the backend.
• Observability: dashboards can separate crash rate by build, hardware revision, reset reason, and signature.
• Regression: fault-injection tests run after changes to startup code, linker layout, RTOS, bootloader, or crash schema.

Engineering Exercise

Design a crash architecture for a Cortex-M RTOS controller with 1 MB internal flash, 256 KB RAM, a hardware watchdog, Ethernet connectivity, and an A/B bootloader. Requirements: the device must recover automatically from a single application fault, retain at least one useful crash across power loss, avoid writing flash repeatedly during a boot loop, and upload diagnostics after networking becomes available.

Define the exception record, retained-RAM layout, flash crash-slot format, commit/integrity scheme, build identifier, watchdog policy, boot-failure counter, A/B rollback criteria, upload retry policy, privacy exclusions, and automated symbolication pipeline. Then specify five fault-injection tests that prove the architecture works.

Knowledge Check

1. Why should the fatal exception path be small and bounded?
2. Why must a handler distinguish MSP from PSP?
3. What advantage does retained RAM have over immediate flash logging?
4. Why does flash crash storage need rate limiting or bounded slots?
5. What is the difference between a compact fault record and a core dump?
6. Why must symbolication use the exact firmware build?
7. Why can repeatedly feeding the watchdog from one timer be a weak health check?
8. What problem does a boot-failure counter solve?
9. Why can a core dump be a security risk?
10. Why should fault-injection testing be part of regression testing?

Answers

1. The system may already be partially corrupted; complex allocation, drivers, filesystems, or blocking operations can fail again and destroy the original evidence.
2. The exception frame may be on different stacks depending on whether thread or handler context was active; reading the wrong stack produces invalid crash state.
3. It can preserve a small record quickly without erase latency or flash wear while the system is unstable.
4. A rapid crash loop can wear flash, overwrite the first useful event, or fill storage with identical records.
5. A compact record captures selected registers and metadata; a core dump preserves broader structured register and memory state for offline debugging.
6. Compiler options, link layout, libraries, and source changes move addresses between builds, so the wrong ELF can generate false source locations.
7. The timer can keep running while critical tasks are deadlocked or no longer making useful progress.
8. It detects repeated failure to reach a stable boot state and enables rollback, recovery mode, or service intervention instead of infinite resets.
9. RAM may contain credentials, keys, tokens, user data, or proprietary information.
10. Startup code, linker changes, RTOS updates, and crash-schema changes can silently break the failure path even when normal operation still passes tests.

Elementary Conclusion

The simplest rule is: design the crash path before the crash happens. Save the processor state, prove which firmware build produced it, protect the record across reset, recover through a controlled policy, and test the entire pipeline by deliberately causing known failures. Good post-mortem architecture turns rare field crashes into structured engineering data.

Previous Related Lessons

• OSFTC.010: Firmware Crash Logs and Fault Recovery
• OSFEC.009: CAN Bus Engineering
• OSFEC.005: Bootloader Design and Firmware Update Architecture
• OSFEC.004: Linker Scripts and Startup Code
• OSFEC.003: Hardware Debugging with JTAG, SWD, OpenOCD, and GDB

Primary References

• Arm — Cortex-M Resources and Fault Exception References
• Zephyr Project — Core Dump
• Zephyr Project — Coredump APIs
• GNU GDB — Backtraces
• Memfault — Cortex-M HardFault Debugging

BitcoinVersus.Tech Editor’s Note:
We volunteer daily to ensure the credibility of the information on this platform is Verifiably True. If you would like to support to help further secure the integrity of our research initiatives, please donate using the QR code provided

For best QR-code scanning results, please scan this donation code from a desktop or larger screen. Mobile screens may prevent camera recognition.

BitcoinVersus.Tech provides technical education and analysis for informational purposes only.