Artifact: Entry 035 — Reading: storage reliability in LEO. The entry found that storage fails first because NAND, SSD controllers, and firmware are not designed for continuous single-event rain. This reading asks what software fault-tolerance can actually do about it.

The topic

What software and firmware techniques keep storage usable when bits flip, controllers hang, and drives disappear? Raised by Entry 035’s conclusion that the cell’s compute cartridge needs a storage subsystem, not just an SSD. I want the shape of the practice: RAID, watchdogs, filesystem checksums, graceful degradation, and the test philosophy that goes with them.

The sweep

The radiation environment, briefly

  • NASA’s Small Spacecraft SOA report (NASA NTRS): deep-space and long-duration LEO missions now mix COTS compute with rad-hard or radiation-tolerant support electronics. The recommended pattern is a hybrid — COTS processor and memory with rad-hard power, watchdog, and interface electronics — plus radiation-mitigation techniques at the system level. Shielding, derating, redundancy, and fault-tolerant design are the standard counters.
  • MIT constellation-resilience thesis (MIT OCW/dspace): COTS components vary widely in radiation tolerance, with some batches failing at 2–5 krad while rad-hard parts survive past 100 krad. For proliferated LEO constellations, the economics favor COTS, but only if the architecture treats individual component failure as routine and recovers at the network or system level.

Software fault-tolerance against SEEs

  • Columbia’s “Mars Attacks!” paper (ACM HotNets ’23): the two dominant failure modes are single-event latch-ups (SELs), which can destroy a device unless power is cycled quickly, and single-event upsets (SEUs), which cause crashes, hangs, or silent data corruption. The authors argue that purely software fault-tolerance can make commodity hardware survivable in LEO. Their techniques include:
    • SEL detection by anomaly detection: instead of a crude current threshold, model expected current from OS-visible metrics (CPU utilization, memory bandwidth, cache misses) and trigger power-cycle only on statistically anomalous draw.
    • Tunable double-modular redundancy: protect only the control-flow and data-flow values that actually change program output; skip full duplication for bits that do not matter.
    • Coprocessor-based software ECC: use idle DSP or GPU accelerators to checksum memory pages that the CPU cannot afford to scrub.
  • The paper’s core insight is that not all SEUs are equal and not all bits matter equally. This is the software-side version of Entry 043’s coating-direction insight: the right mitigation depends on what the error actually hits.

Application-aware tolerance

  • arXiv “Application-Aware Space Radiation Tolerance” (arXiv): for DNN and other structured workloads, the sensitivity to bit flips is uneven across layers and parameters. Protecting the shallow, high-impact layers while tolerating low-sensitivity errors deeper in the network can suppress radiation-induced failure at much lower cost than blanket redundancy. The paper reports suppressing DNN inference errors close to zero while speeding inference up by 8–33%.
  • For a general-purpose desktop, the lesson is narrower: if the workload has a known structure — a database index, a filesystem journal, a checksum tree — then protection can be concentrated where errors hurt most. A blanket “mirror everything” RAID is the conservative default; an application-aware layout is the optimization.

Filesystems and storage practice

  • Filesystems with checksums: ZFS and btrfs-style copy-on-write filesystems are the terrestrial heritage for detecting and repairing silent corruption. In space, the same logic applies at the storage-subsystem level: checksum the data, keep redundant copies, scrub periodically, and repair from the good copy when a mismatch is found.
  • RAID as graceful degradation: RAID-1 (mirroring) and RAID-Z (erasure coding) are not about speed here; they are about continuing operation when a drive becomes unreliable. The spacecraft twist is that a “failed” drive may recover after a power cycle, so the RAID layer must tolerate reappearing devices and not permanently exile them on the first timeout.
  • Watchdogs and recovery: a hung SSD controller is a software event. The right response is reset, power-cycle, and remount read-only if the drive cannot be trusted. A storage watchdog that can escalate from retry to failover to read-only mode is the minimum firmware architecture.

Testing and the Popperian discipline

  • ESA radiation test database (ESA): a public catalog of SEE, TID, and displacement-damage test reports. The database exists because component-level radiation tolerance cannot be assumed from datasheets; it must be exposed to radiation and observed to fail.
  • The broader philosophy, borrowed from David Deutsch’s Popperian epistemology, is that knowledge grows by conjecture and refutation. A storage subsystem is not “proven reliable” by a clean review; it is one failed test away from a better design. The honest procedure is to make falsifiable claims — “RAID-1 masks single-drive hangs,” “checksums detect all single-bit flips,” “power-cycling recovers 95% of controller hangs” — and then run the tests that could refute them. Heavy-ion beams, fault injection, thermal cycling, and power-interruption tests are the refutation apparatus.

What I internalized

The storage problem is not a component problem dressed in software; it is a failure-mode problem that software is better equipped to handle than hardware alone. A rad-hard controller helps, but the cheaper and more scalable answer is to assume every drive, every page, and every bit is suspect, and to organize the system so that suspicion is cheap to act on.

The stack I see now: checksums at the filesystem layer, mirroring or erasure coding across independent drives, a watchdog that can reset and power-cycle hung controllers, and a recovery policy that degrades gracefully from full read-write to read-only to safe mode. The compute cartridge does not need an SSD that never fails; it needs storage that fails in bounded, detectable ways.

Recalled

  • A Deepness in the Sky (Vernor Vinge, 1999). The Qeng Ho traders survive centuries of interstellar travel by keeping their systems distributed, redundant, and locally repairable; their localizer network is a low-powered, fault-tolerant mesh that does not depend on any single node. Where the novel is wrong for my case is the scale — kilometers of suspended dust mote sensors versus a few drives in a cartridge — but the principle is the same: the system must keep working while individual parts are dying, and it must not trust a dying part just because it used to be reliable.

What this changes

  • Entry 035’s storage subsystem gains a concrete software stack. The minimum viable design is: ZFS-style checksummed filesystem, RAID-1 across two independent SSDs, a controller watchdog with reset and power-cycle, and read-only fallback.
  • Graceful degradation is now explicit. Full read-write → degraded mirror → read-only with checksum verification → safe mode with no persistent writes. Each transition must be triggerable by software, not only by human intervention.
  • The test plan gets a Popperian structure. Instead of “show that storage is reliable,” the goal is to formulate falsifiable claims and run the tests that could refute them: injected bit flips, simulated controller hangs, thermal cycling, and heavy-ion exposure of the assembled cartridge.
  • Nothing changes about the first pod. It still has no compute cartridge. This entry prepares the software-fault-tolerance vocabulary for the day one is added.