FAULTLINE SYSTEMS LAB / 01

A healthy service
can still be wrong.

Break the workflow. Compare the repairs. Inspect the evidence.

The engineering behind it Recorded locally · Synthetic data

POSTGRESQL + PYTHON / Replay · seed 103

REPORTED REVENUE$2,778.00Incident captured

EXPECTED FROM VALID EVENTS$2,087.0017 deliveries · 12 valid business events

RECONCILIATION GAP+$691.002 / 4 invariants pass

01 / INGEST

Event stream

17 deliveries

HTTP → durable inbox

02 / PROCESS

Worker

Legacy writes

Separate write and acknowledgement

03 / STORE

Revenue ledger

17 entries

17 deliveries acknowledged

04 / VERIFY

Business truth

Checks fail

Full records, independently checked

ONE INCIDENT. THREE BRANCHES.

Which repair actually works?

Each starts from an identical copy of the captured database state.

WHAT WENT WRONG

A repeated delivery creates a second financial effect. The worker finishes successfully. The reported revenue is wrong.

Follow the records.

An event is unique within its tenant. Two valid events may belong to the same order.

Posted ledger entries at Incident captured
Tenant / event	Order	Posted USD	Check
cedar`evt-0001`	ord-0000	$316.00	Duplicate
harbor`evt-0000`	ord-0000	$241.00	Duplicate
cedar`evt-0000`	ord-0000	$118.00	Matches
cedar`evt-0005`	ord-0002	-$2.00	Duplicate
cedar`evt-0004`	ord-0002	$68.00	Duplicate
harbor`evt-0001`	ord-0000	$47.00	Matches
cedar`evt-0004`	ord-0002	$68.00	Duplicate
cedar`evt-0001`	ord-0000	$316.00	Duplicate
harbor`evt-0002`	ord-0001	$70.00	Matches
cedar`evt-0002`	ord-0001	$343.00	Matches
harbor`evt-0003`	ord-0001	$203.00	Matches
cedar`evt-0003`	ord-0001	$244.00	Matches
harbor`evt-0004`	ord-0002	$181.00	Matches
cedar`evt-0004`	ord-0002	$68.00	Duplicate
harbor`evt-0005`	ord-0002	$258.00	Matches
cedar`evt-0005`	ord-0002	-$2.00	Duplicate
harbor`evt-0000`	ord-0000	$241.00	Duplicate

OPEN THE EVIDENCE

A result you can trace.

Download this run (JSON)

What the model actually saw

{
  "facts": [
    {
      "id": "OBS-1",
      "kind": "health",
      "database": "reachable",
      "worker": "available"
    },
    {
      "id": "OBS-2",
      "kind": "delivery",
      "received": 17,
      "acknowledged": 17,
      "pending": 0
    },
    {
      "id": "OBS-3",
      "kind": "projection",
      "entries": 17,
      "reported_minor": 277800,
      "duplicate_effects": [
        {
          "tenant": "cedar",
          "event_id": "evt-0001",
          "entries": 2
        },
        {
          "tenant": "harbor",
          "event_id": "evt-0000",
          "entries": 2
        },
        {
          "tenant": "cedar",
          "event_id": "evt-0005",
          "entries": 2
        },
        {
          "tenant": "cedar",
          "event_id": "evt-0004",
          "entries": 3
        }
      ]
    },
    {
      "id": "OBS-4",
      "kind": "contract",
      "invalid_envelopes": [],
      "quarantined": 0
    }
  ],
  "sample": [
    {
      "seq": 1,
      "tenant": "cedar",
      "event_id": "evt-0001",
      "order_id": "ord-0000",
      "amount_minor": 31600
    },
    {
      "seq": 2,
      "tenant": "harbor",
      "event_id": "evt-0000",
      "order_id": "ord-0000",
      "amount_minor": 24100
    },
    {
      "seq": 3,
      "tenant": "cedar",
      "event_id": "evt-0000",
      "order_id": "ord-0000",
      "amount_minor": 11800
    },
    {
      "seq": 4,
      "tenant": "cedar",
      "event_id": "evt-0005",
      "order_id": "ord-0002",
      "amount_minor": -200
    },
    {
      "seq": 5,
      "tenant": "cedar",
      "event_id": "evt-0004",
      "order_id": "ord-0002",
      "amount_minor": 6800
    },
    {
      "seq": 6,
      "tenant": "harbor",
      "event_id": "evt-0001",
      "order_id": "ord-0000",
      "amount_minor": 4700
    }
  ],
  "revision": 18
}

Action receipts for this branch

[
  {
    "operation": "rebuild_projection",
    "accepted": true,
    "request_id": "928dd342f36944ca8a0146a34a8fb0a9",
    "status": "complete",
    "elapsed_ms": 95.555
  }
]

Captured 2026-09-11 UTCBackend revision 849ebac937fcLocal execution; this page replays saved results

THE FULL CASE STUDY

What the experiment taught us.

Architecture, all 30 model attempts, automated tests, and the reproducible source.