Operational

OPERATIONS · RUNBOOK INDEX

Repeatable response patterns with explicit stop conditions.

These runbooks define entry signals, validation, safe containment, verification, and escalation. They provide operating structure without publishing credentials or provider-specific emergency paths.

Owner
Global Reliability
Version
1.3
Reviewed
05 Aug 2026
Review trigger
After material incident

Runbook selection

IDEntry signalFirst validationEscalation owner
RB-AUTH-01Repeated login or MFA failuresConfirm source scope, rate response, and identity statusSecurity Assurance
RB-API-02Elevated 5xx or readiness failureCompare origin health and local API readinessPlatform SRE
RB-XFER-03Transfer blocked or lag above objectiveVerify route, region state, and approved maintenanceData Operations
RB-INT-04Integration registration check failsConfirm registry alias and last successful checkService owner
RB-SEC-05Possible unauthorized actionPreserve request IDs and audit rangeSecurity Incident Commander

RB-AUTH-01 · Authentication failure concentration

  1. Determine whether failures affect one identity, one source boundary, or the complete service.
  2. Confirm whether responses are 401, 403, or 429; they imply different control paths.
  3. Do not reset credentials or disable MFA before validating ownership through the registered support route.
  4. Preserve request identifiers and the relevant UTC window.
  5. Escalate immediately if a successful session follows unexplained failure concentration.

Success condition: expected identities can authenticate, rate windows recover normally, and no unauthorized session remains active.

RB-API-02 · API readiness or error-rate degradation

  1. Compare /health, /ready, and /api/v1/status.
  2. Separate static-origin availability from local API readiness.
  3. Check the last deployment and whether the failure began inside its validation window.
  4. Prefer rollback to an unreviewed live edit when the release is the confirmed cause.
  5. Verify authentication, one read route, and one authorized write route after recovery.

RB-XFER-03 · Blocked transfer or replication lag

Confirm source and target are distinct, both regions are available, and the data space permits the route and retention behavior. Pause retry loops when a conflict or policy denial is deterministic. A rising queue with stable throughput requires different action from zero throughput with provider or network failure.

RB-SEC-05 · Suspected unauthorized action

Preserve before changing

Record actor, resource, action, outcome, request ID, earliest time, latest time, and affected session. Do not copy session cookies, factor values, or active secrets into the incident record.

  1. Validate the audit event against the resource's current state.
  2. Revoke the affected server-side session when containment is authorized.
  3. Identify whether the failure involved credentials, role assignment, CSRF/origin handling, or an operational approval.
  4. Limit containment to the affected boundary where evidence supports it.
  5. Move to the full security incident process.

Closure requirements

Every runbook execution records the entry signal, timeline, decisions, changes, request IDs, validation, remaining uncertainty, owner, and follow-up actions. A runbook is not closed solely because an alert cleared; the declared success condition must be observed.