Module 11 · Platform Operations · 11C

Resilience & Replay

Can we recover without double-counting?

Try this

1Replay RUN-042 once
2Replay again
Expected: First replay inserts the missing 23 keys; second inserts 0 — total stays 36.

3/4

Controlled fault injection

Run failure

HTTP 503 → 3 retries with exponential backoff (30s, 2m, 8m)

Partial response

Pagination truncated → backfill missed window from raw store

API delivery failure

Gateway 502 → queued, redelivered with same idempotency key

DR event

Primary region unavailable → restore from backup within 1 business day (scenario)

Idempotent replay of RUN-042 · upsert key = run_id + listing_key

Original

of 36

Replays

Unique listings

Latest

RUN-042

stale

No replays yet.

Reconciliation

before: 13 unique (+4 dup collapsed)
replay #1: +23 inserted → 36
replay #2+: +0 inserted → 36   (idempotent)

Twice-yearly DR test history (fictional)