Tell on-call what actually shipped

When something breaks shortly after a deploy, the person on call spends their first ten minutes establishing what shipped. That is ten minutes of an incident spent on a question with a written answer.

Post what shipped, every time

Before any alerting logic, just narrate deploys into the channel. It is a small thing that changes incident response more than it sounds.

When a deploy to production completes, post to the on-call channel:

  what version shipped and when
  the pull requests included, one line each
  which areas of the system they touch
  anything in this deploy that touches auth, payments, or the database
    schema, called out separately

Keep it short. Nobody reads a changelog during an incident.

The line about auth, payments and schema is the part on-call actually uses. Most deploys are irrelevant to most incidents, and a list of nine pull requests is a list to be skimmed. Knowing that this one touched the payment path turns a search into a first guess, which is the whole value.

Then correlate errors with it

For 30 minutes after each deploy, watch error rates.

If errors rise materially against the pre-deploy baseline, post to the
on-call channel: what changed, when it started, which errors, and the
deploy you think is implicated.

Say you think it is implicated. Do not assert causation. A rise after a
deploy is a correlation and the responder needs to know both facts, not a
conclusion.

If nothing rises, say nothing.

The wording matters more than the detection

"Errors up 300% since deploy abc123" and "deploy abc123 broke checkout" produce different investigations. The second sends someone to read a diff; if the real cause was a third party failing at the same moment, they lose fifteen minutes disbelieving the code.

Correlation stated as correlation keeps the responder's options open.

Do not page on this

Post, do not page. This is context for someone already responding, or a nudge for someone glancing at a channel. Paging on a correlation produces false pages, and false pages are how a rota stops trusting its alerting.

Whatever pages today should keep paging. This sits beside it.

Include the rollback

One line in the deploy message with the command or the link to revert. At 3am, finding the rollback procedure is another few minutes of the same wasted kind.

What good looks like

On-call opens the channel and can already see what shipped, whether it touched anything dangerous, and how to undo it.

The measure is time-to-first-hypothesis during your next incident. If it is still spent working out what changed, the deploy message is not being read, which usually means it is too long.