# Degradation runbook

Each failure, what Mirror does by itself, and what you do. The rule behind every row: **Mirror degrades toward not clearing and toward telling you, never toward a silent pass.** Each row is exercised by `server/degradation.test.js` or the named test.

| Failure | What Mirror does | What you see | What you do |
|---|---|---|---|
| **The operator is unreachable** | Nothing changes inside the bank. The bander has no network; it bands and signs legs offline. | Your submission client gets no answer | Keep the signed legs and resubmit when the operator is back. A resubmission is the same verdict, never a second one. |
| **The EU operator object is unreachable, but the edge is up** | The edge refuses the request `503 unavailable`. It never processes it at the edge or anywhere else, so the residency commitment holds through the outage. (`server/residency.test.js`) | `503`, `code: "unavailable"` | Resubmit with backoff: the same legs give the same verdict. |
| **The operator's store refuses a write** | The verdict is still signed and returned | `ledger: "NOT appended: ..."` on the response | Keep the signed verdict. Resubmit the same legs later to append it (same judgement, counted once). |
| **A governance event cannot be recorded** | Refused `503 store_unwritable`; nothing changes anywhere | 503 | Retry the same signed event |
| **The store is unreadable at start-up** | The operator starts with no roster, so no leg authenticates and nothing clears | `undetermined` verdicts | Restore the store (see [RTO / RPO](/rto-rpo.md)) |
| **The operator's signing key is unavailable** | `503 no_key`: never an unsigned verdict | 503 | The operator restores its key |
| **No independent witness has answered** | Agreeing legs stay `undetermined` (`insufficient_independence`) | `undetermined` | Wait for the terminal's bulletin or measurement, or use a ratified relay |
| **A witness key is compromised** | At quorum suspension, each affected member is pushed its signed recall notice | `recall.notice` webhook; `/recall/notice` | Re-review the clears marked "loses its footing" first |
| **Your HSM is unreachable** | The bander stops before reading a row. There is no fallback to a key file and no batch. | Exit code 2, `custody: the HSM/KMS did not answer` | Restore HSM access and rerun. Banding is deterministic, so the rerun is the same batch. (`server/bander-ha.test.js`) |
| **One bander host dies mid-run** | The other host bands the same file to byte-identical commitments | — | Run it there. The operator counts one verdict either way. |
| **Your webhook endpoint is down** | Retried at 30 s, 2 m, 10 m, 1 h, 6 h and 24 h, then `dead`. A `410` pauses the subscription. | `/webhooks/list` shows the state | Fix the endpoint, then `/webhooks/redeliver`. The ledger was never affected. |
| **Clock skew** | Freshness checks refuse; verdicts and key rotation do not depend on the clock | Refused sign-ins or deliveries | Fix the clock. `POST /time` bounds the offset ([clock discipline](/clock-discipline.md)). |
| **A log fork is detected** | Gossip produces two signed heads that cannot both be true | `/anchor/gossip` | Stop relying on new heads; keep your verdicts; see the [log policy](/transparency-log-policy.md) |