Incident response runbook

This is the operator checklist for a security, data-integrity, deployment, or availability incident. Preserve evidence and customer data before attempting a cleanup. Never paste secrets, private keys, raw environment files, or unredacted customer logs into tickets or chat.

Severity

One person is incident commander, one is operations lead, and one owns customer communications when staffing allows. Record UTC times, commands, affected resources, decisions, and evidence locations in a private incident log.

First 15 minutes

1. Confirm the symptom from a second signal: API health, audit history, Docker state, provider status, or a customer reproduction.
2. Declare severity and scope: self-hosted or managed, customer IDs, repositories, environments, servers, connectors, builds, and time range.
3. Freeze the smallest unsafe mutation surface. Revoke a token/session/connector, disable a provider webhook, pause build workers, or remove approval rights; do not shut down unrelated workloads.
4. Preserve the hub logs, edge request IDs, audit events, database snapshot, relevant connector command envelopes, image digests, and deployed state.
5. Establish the last known good release, backup checksum, deployment, and connector certificate state.
6. Send an initial notice that states impact, current containment, and the next update time. Do not speculate about root cause.

Containment playbooks

Leaked dashboard, service-account, or provider credential

• Revoke the affected session, service account, PAT, or provider token.
• Rotate downstream credentials before issuing a replacement ReaperCI token.
• Review audit events and provider webhook delivery history from the earliest possible exposure time.
• Inspect builds, artifacts, images, promotions, approvals, server actions, and connector commands attributed to the identity.
• Rebuild trusted images from a known-good commit after dependency and pipeline review. Do not merely retag an image of uncertain provenance.

Suspected master-key exposure

• Treat every encrypted secret in that instance as exposed if the attacker may also have copied the data store.
• Stop new builds and deployments, retain read-only access for evidence, and rotate repository, provider, server, connector, registry, OIDC, and notification credentials at their source.
• Generate a new master key and run the documented key rotation during a maintenance window. Keep the prior key only in the sealed incident evidence.
• Produce a fresh backup and invalidate older backups according to the response and legal retention decision.

Malicious or escaped build

• Cancel queued/running builds for the repository and disable its webhook.
• Stop the affected BuildKit worker. Capture worker, network, and image metadata before replacement; do not reuse its cache as trusted input.
• Inspect whether the worker could reach anything beyond the documented private registry/hub network. Rotate any credentials granted to the build.
• Provision a clean worker, rebuild from the reviewed commit without cache, and compare image digest and SBOM before resuming.

Unsafe or failed production deployment

• Reject other pending deployments to the environment and disable automatic promotion.
• Use Deploys > Inspect or reaperci deploys inspect ID to capture live state.
• Review rollback-target, confirm the exact prior successful deployment ID, and execute the explicit rollback. Protected environments still require a human approval.
• Verify the configured health endpoint and user-visible behavior. A green container alone is not recovery.
• If rollback is unsafe, restore application data using that application's own runbook; ReaperCI does not infer database rollback safety.

Connector compromise or unexpected command

• Revoke the connector session in Settings > Connectors. Revocation is enforced at the next hub request.
• On the VPS, stop the connector service and preserve its state directory and service logs. Rotate the Unix account or Docker permissions if they may have been used outside the connector.
• Review command IDs, operation names, requested actors, encrypted-payload use, results, and audit events. A command absent from the allowlist is evidence of a protocol or binary integrity failure.
• Reinstall a signed connector binary and enroll with a new one-time token only after the host is trusted again.

Disk full, registry failure, or corrupt data volume

• Stop new builds and registry uploads. Record filesystem usage and the largest ReaperCI paths without deleting data.
• Expand storage or move the complete volume according to the hosting runbook. Do not run registry garbage collection while the hub is serving uploads.
• If corruption is suspected, restore the matching data backup and separately retained master key on an isolated host first. Verify repositories, registry pulls, build history, artifacts, and a non-production deployment.
• Run offline registry garbage collection only after the recovered state is backed up and the hub is stopped.

Recovery verification

Recovery is complete only when the API and browser are healthy, authentication works, affected credentials are rotated, the audit trail is preserved, a clean build can push and pull an image, a non-production deployment passes its health check, production state is inspected, and backup/restore evidence is current. Keep heightened monitoring through at least one normal build and deployment cycle.

Customer communication

Every update includes: confirmed impact, affected time window, current state, actions customers must take, and next update time. Security notifications state which credentials or data classes were involved without exposing their values. Publish a final incident report only after legal/security review when personal data or customer secrets may be involved.

Post-incident work

Within five business days, document the timeline, contributing conditions, detection gap, containment and recovery decisions, customer impact, and durable corrective actions. Each action has an owner, due date, and verification method. Add a regression test or restore drill when software or procedure can prevent a repeat.