On-Call & Runbooks

On-call exists so customers get help fast. It only works long term if it's sustainable for the people carrying the pager: few pages, every page actionable, and time set aside to make next week quieter than this one.

Rotations

The team that ships the service carries its pager. Ownership and on-call are the same thing.
One-week shifts, primary and secondary, handed off on a weekday morning, never on a Friday evening.
At least 6 people per rotation. Fewer than that and people are on call too often to recover between shifts.
Follow the sun when you can. Nights are the exception, not the job.
On-call is the shift's main work. Feature work is a bonus. The rest of the shift goes to toil reduction and alert hygiene.

Severity Levels

Severity	Definition	Response
SEV1	Critical journey down, money moving incorrectly, or data exposed	Page immediately, incident commander, status page, all hands as needed
SEV2	Major degradation, or an SLO burning fast	Page, fix during the shift
SEV3	Minor or contained impact	Ticket, fix within the week

Declare early and downgrade later. A SEV1 that turns out to be a SEV3 costs an hour. A SEV3 that was really a SEV1 costs customers.

During an Incident

One incident commander. They coordinate and decide; they don't debug. Everyone else works the problem.
Mitigate first, diagnose later. Roll back, fail over, flip the kill switch. Understanding why can wait until customers are fine.
One channel, one timeline. Decisions and actions go in the incident channel with timestamps. It becomes the postmortem timeline.
Communicate on a schedule. Status updates every 30 minutes for SEV1, even when the update is "still investigating".

Escalation

Primary acknowledges within 5 minutes for a page. No acknowledgement escalates to secondary automatically, then to the engineering manager.
Escalating is never a failure. Pulling in the database owner at 3am is cheaper than guessing for two hours.
Every service lists who to escalate to, in the runbook, not in someone's memory.

Handoffs

Every shift ends with a short written handoff:

## Handoff 2026-10-19 to 2026-10-26

Pages: 3 (2 actionable, 1 noisy: see ticket)
Open incidents: none
Ongoing: payments-db replica lag alerts after Tuesday's index build, watching
Changes to know about: processor failover flag is ON for EU traffic
Toil worth fixing: manual webhook redelivery for disabled endpoints

Runbooks

Every alert links to a runbook. An alert without one doesn't get to page anyone.

# Alert: PaymentsAuthorizationBurnRateHigh

## What it means
Authorization failures are spending the 30-day error budget at 14.4x or more.

## Impact
Customers can't pay. Merchants see failed checkouts.

## Triage
1. Check the authorization rate dashboard by processor and card brand.
2. One processor? Check its status page, then step 1 of Mitigation.
3. All processors? Check the most recent deploy and roll it back.

## Mitigation
1. Fail over the affected processor: flip the `processor-failover` flag.
2. ...

## Escalation
Payments team secondary, then the processor's support line (in the vendor doc).

Runbooks are kept next to the code, reviewed in pull requests, and updated after every incident that found them wrong or missing.

Toil

Toil is manual, repetitive, automatable work that grows with the system and leaves nothing behind. The Google SRE Book caps it at 50% of an SRE's time, and that's a ceiling, not a target.

Track it. Every manual intervention on shift gets a tally in the handoff.
Fix the top item every rotation. Automate it, or make the alert that caused it go away.
Delete noisy alerts. An alert that's acknowledged and ignored twice gets fixed or removed.

References

In practice

Google SRE Book: Being On-Call, rotation size and load
Google SRE Workbook: On-Call, handoffs and sustainable rotations
Google SRE Book: Managing Incidents, incident roles
Google SRE Workbook: Incident Response, running an incident
Google SRE Book: Eliminating Toil, the 50% toil cap
PagerDuty: Incident Response Documentation, severity levels and the incident commander role

Rotations

The team that ships the service carries its pager. Ownership and on-call are the same thing.

One-week shifts, primary and secondary, handed off on a weekday morning, never on a Friday evening.

At least 6 people per rotation. Fewer than that and people are on call too often to recover between shifts.

Follow the sun when you can. Nights are the exception, not the job.

On-call is the shift's main work. Feature work is a bonus. The rest of the shift goes to toil reduction and alert hygiene.

Severity Levels

Severity

Definition

Response

SEV1

Critical journey down, money moving incorrectly, or data exposed

Page immediately, incident commander, status page, all hands as needed

SEV2

Major degradation, or an SLO burning fast

Page, fix during the shift

SEV3

Minor or contained impact

Ticket, fix within the week

Declare early and downgrade later. A SEV1 that turns out to be a SEV3 costs an hour. A SEV3 that was really a SEV1 costs customers.

During an Incident

One incident commander. They coordinate and decide; they don't debug. Everyone else works the problem.

Mitigate first, diagnose later. Roll back, fail over, flip the kill switch. Understanding why can wait until customers are fine.

One channel, one timeline. Decisions and actions go in the incident channel with timestamps. It becomes the postmortem timeline.

Communicate on a schedule. Status updates every 30 minutes for SEV1, even when the update is "still investigating".

Escalation

Primary acknowledges within 5 minutes for a page. No acknowledgement escalates to secondary automatically, then to the engineering manager.

Escalating is never a failure. Pulling in the database owner at 3am is cheaper than guessing for two hours.

Every service lists who to escalate to, in the runbook, not in someone's memory.

Handoffs

Every shift ends with a short written handoff:

## Handoff 2026-10-19 to 2026-10-26 Pages: 3 (2 actionable, 1 noisy: see ticket) Open incidents: none Ongoing: payments-db replica lag alerts after Tuesday's index build, watching Changes to know about: processor failover flag is ON for EU traffic Toil worth fixing: manual webhook redelivery for disabled endpoints

Runbooks

Every alert links to a runbook. An alert without one doesn't get to page anyone.

# Alert: PaymentsAuthorizationBurnRateHigh ## What it means Authorization failures are spending the 30-day error budget at 14.4x or more. ## Impact Customers can't pay. Merchants see failed checkouts. ## Triage 1. Check the authorization rate dashboard by processor and card brand. 2. One processor? Check its status page, then step 1 of Mitigation. 3. All processors? Check the most recent deploy and roll it back. ## Mitigation 1. Fail over the affected processor: flip the `processor-failover` flag. 2. ... ## Escalation Payments team secondary, then the processor's support line (in the vendor doc).

Runbooks are kept next to the code, reviewed in pull requests, and updated after every incident that found them wrong or missing.

Toil

Toil is manual, repetitive, automatable work that grows with the system and leaves nothing behind. The Google SRE Book caps it at 50% of an SRE's time, and that's a ceiling, not a target.

Track it. Every manual intervention on shift gets a tally in the handoff.

Fix the top item every rotation. Automate it, or make the alert that caused it go away.

Delete noisy alerts. An alert that's acknowledged and ignored twice gets fixed or removed.