Make the failure visible.
Make the repair provable.

The runbook won.
The verifier explained why.

The explicit runbook repaired all 15 of 15 scenario variations. The local Qwen3 4B model repaired 20 of 30 attempts. No interrupted-write attempt repaired the ledger: 8 abstained and 2 restarted the worker, leaving a duplicate effect.

The important behavior is visible in the lab: an accepted action can still produce incorrect records. The evaluator reports that failure independently of the model’s explanation.

Three known synthetic fault families, five held-out input variations each, two model attempts per variation. Repeated attempts are correlated; this is a bounded experiment, not a general reliability estimate.

The database is real.
So are the mistakes.

A separate HTTP ingestion process persists fictional sale and refund events in PostgreSQL. A worker turns those deliveries into a revenue ledger. Its legacy mode deliberately permits duplicates, splits writes from acknowledgements, and skips the unit contract.

1. 01

   Capture the incident

   Repeated input, a real worker process exit, or an incompatible envelope creates a measurable discrepancy. The original deliveries and ledger state are retained.

2. 02

   Fork the same state

   Restart, runbook, and model each receive a separate copy. SHA-256 fingerprints confirm matching state at the fork and an unchanged source afterward.

3. 03

   Constrain the action

   The model selects one operation from a closed catalog. A host-owned broker enforces the run identity, action budget, deadline, cancellation, and receipt-based retries. The model receives no shell, SQL, or arbitrary network tool.

4. 04

   Verify the business records

   An independent evaluator compares identities, exact amounts, completeness, duplicate effects, and quarantine against truth generated before fault injection. Matching totals alone cannot pass a corrupted ledger.

Safe processing commits the tenant-scoped event identity, ledger effect, and acknowledgement together. An incompatible record remains available for review. Repair rebuilds only the disposable branch’s projection.

Publish every attempt.

The protocol, action catalog, prompt, and code were fixed before the held-out execution. The baseline and model received the same observations and used the same executor. The three examples in the lab always use seed 103, attempt 0; they were not selected for a successful model outcome.

Model selection latency was 1.29 seconds at the median and 1.65 seconds at p95. These are measured local request times, with linearly interpolated percentiles; they exclude recovery execution. Hardware: RTX 2080 SUPER, 8 GB VRAM, approximately 125 GiB system RAM. No paid inference service was used.

All 30 planned attempts completed without a model request error in this evaluation. An earlier development run had a cold-start timeout; it remains documented separately. Model settings: temperature 0, 4,096-token context, 240-token completion cap, 60-second request timeout.

Test the failures
that matter.

16 backend tests passed against PostgreSQL in 4.42 seconds. They exercise real process exits and concurrent workers, as well as the contract and authority boundaries.

• Four process exit boundaries, followed by recovery.
• Concurrent workers, tenant-scoped identity, multiple valid events on one order, and refunds.
• Action retries, interrupted repair receipts, cancellation, deadlines, budget limits, and cross-run denial.
• Conflicting payload quarantine and explicit review of incompatible units.
• A corrupted ledger that balances in total but contains incorrect records.
• HTTP origin, content-type, size, identifier, and envelope rejection.

The browser replay is also checked for agreement with the captured records and tested across desktop and narrow layouts. The detailed release checks are retained alongside the source; physical-device acceptance is separate.

Know what this proves.

The local model is a one-step recovery selector over known operations. The work demonstrates bounded execution, evaluation design, failure analysis, reproducible records, and a usable explanation of the system. It does not establish general incident diagnosis or production readiness.

The public website serves captured synthetic records. Visitors can explore and download them; they cannot trigger private services or run new model calls. The downloadable Python project runs the actual services locally.

There are three known fault families, small datasets, one model, and one workstation. The broker’s controls are tested boundaries within this lab, not an exhaustive security assessment. Observations cite known IDs, but the prose explanation is not independently scored for semantic accuracy.

Miguel selected the project objective and authorized the build. Codex assisted with implementation, documentation, and automated verification. A personal technical walkthrough and interview preparation are still separate work; the project does not claim independently assessed proficiency.

Run the system locally.

The source bundle preserves the evaluated worker, broker, evaluator, fixtures, and tests. Release changes add documentation, export tooling, and a database startup fix. The exporter checks that the evaluated core is unchanged. It requires Python 3.12+, Docker, and an existing local Ollama model for the optional model comparison.

python3 -m venv .venv
.venv/bin/pip install -r requirements.lock
.venv/bin/pip install --no-deps -e .
python3 scripts/lab-db.py start
.venv/bin/python -m unittest discover -s tests -v
.venv/bin/python -m faultline.cli run \
  --output artifacts/my-runs --seeds 17
python3 scripts/lab-db.py stop

Add --model qwen3:4b to execute the local model comparison. The database helper owns only its labelled, disposable lab container and volume.