DocumentsRunning it

Degradation runbook

2 min readSource degradation-runbook.mdsha256 f24056b7a74ec973…

Each failure, what Mirror does by itself, and what you do. The rule behind every row: Mirror degrades toward not clearing and toward telling you, never toward a silent pass. Each row is exercised by server/degradation.test.js or the named test.

FailureWhat Mirror doesWhat you seeWhat you do
The operator is unreachableNothing changes inside the bank. The bander has no network; it bands and signs legs offline.Your submission client gets no answerKeep the signed legs and resubmit when the operator is back. A resubmission is the same verdict, never a second one.
The EU operator object is unreachable, but the edge is upThe edge refuses the request 503 unavailable. It never processes it at the edge or anywhere else, so the residency commitment holds through the outage. (server/residency.test.js)503, code: "unavailable"Resubmit with backoff: the same legs give the same verdict.
The operator's store refuses a writeThe verdict is still signed and returnedledger: "NOT appended: ..." on the responseKeep the signed verdict. Resubmit the same legs later to append it (same judgement, counted once).
A governance event cannot be recordedRefused 503 store_unwritable; nothing changes anywhere503Retry the same signed event
The store is unreadable at start-upThe operator starts with no roster, so no leg authenticates and nothing clearsundetermined verdictsRestore the store (see RTO / RPO)
The operator's signing key is unavailable503 no_key: never an unsigned verdict503The operator restores its key
No independent witness has answeredAgreeing legs stay undetermined (insufficient_independence)undeterminedWait for the terminal's bulletin or measurement, or use a ratified relay
A witness key is compromisedAt quorum suspension, each affected member is pushed its signed recall noticerecall.notice webhook; /recall/noticeRe-review the clears marked "loses its footing" first
Your HSM is unreachableThe bander stops before reading a row. There is no fallback to a key file and no batch.Exit code 2, custody: the HSM/KMS did not answerRestore HSM access and rerun. Banding is deterministic, so the rerun is the same batch. (server/bander-ha.test.js)
One bander host dies mid-runThe other host bands the same file to byte-identical commitments—Run it there. The operator counts one verdict either way.
Your webhook endpoint is downRetried at 30 s, 2 m, 10 m, 1 h, 6 h and 24 h, then dead. A 410 pauses the subscription./webhooks/list shows the stateFix the endpoint, then /webhooks/redeliver. The ledger was never affected.
Clock skewFreshness checks refuse; verdicts and key rotation do not depend on the clockRefused sign-ins or deliveriesFix the clock. POST /time bounds the offset (clock discipline).
A log fork is detectedGossip produces two signed heads that cannot both be true/anchor/gossipStop relying on new heads; keep your verdicts; see the log policy