1. The bug that had to be survived
Andy Weir’s The Martian returns as the recalled work because Mark Watney’s life is a series of workarounds for things that break in ways no one anticipated. Software on the desktop will break the same way. The test matrix is the inventory of anticipated-unanticipated faults that the platform must survive.
This entry defines the software fault injection test matrix.
2. Test setup
The test uses a flight-representative software stack running on representative hardware or a high-fidelity simulator. Faults are injected into individual tasks, messages, resources, and timing. The platform is expected to detect, contain, and recover without human intervention for faults within the matrix. Task state, message traffic, resource usage, and recovery actions are recorded continuously.
3. The test matrix
| Case | Scenario | Conditions | Expected outcome |
|---|---|---|---|
| 1 | Application crash | A non-critical application exits unexpectedly | Supervisor restarts the application; critical services continue. |
| 2 | Application hang | A critical application stops responding | Watchdog or health monitor detects hang and restarts the task. |
| 3 | Corrupted message | A message on the software bus has invalid data | Subscribers reject or sanitize the message; no task crashes. |
| 4 | Message flood | A faulty application publishes messages faster than consumers can handle | Rate limiting or backpressure prevents queue overflow; critical traffic is prioritized. |
| 5 | Memory leak | A task slowly leaks memory | Memory monitor detects trend and triggers restart before exhaustion. |
| 6 | CPU overload | A runaway task consumes excessive CPU | Scheduler protects critical tasks; overload is logged and the task is throttled or restarted. |
| 7 | Priority inversion | A low-priority task holds a resource needed by a high-priority task | Priority inheritance or ceiling protocol prevents indefinite blocking. |
| 8 | Incompatible update | A staged update has a version conflict | Verification rejects the update before activation; previous image remains active. |
| 9 | Failed rollback | A new image fails to boot | Bootloader selects the previous image; ground is notified. |
| 10 | Cascading fault | A recovery action triggers a secondary fault | Supervisor detects escalation and enters a safe, stable mode. |
4. What is measured
For each case the test records:
- Detection time and classification accuracy.
- Whether critical services remained available.
- Whether the fault was contained or propagated.
- Recovery time and method.
- Whether data integrity was preserved.
- Whether the platform returned to a stable state.
- Whether ground could reconstruct the event from logs.
5. What this changes
- The software fault injection test has a concrete matrix.
- The matrix links back to the flight software reading, autonomous update wondering, and earlier autonomy arcs.
- The next entry will define pass and fail criteria.