Keep runbooks from going stale

Runbooks rot in a specific way. They are written accurately, the infrastructure changes, and nobody updates them because the person who made the change did not know a runbook mentioned it.

You discover this at 3am, following instructions for a system that no longer works that way.

Get the runbooks somewhere readable

Into Documents. Whatever state they are in.

If several are stale enough to be misleading, that is worth knowing before you automate anything, and the first pass below will tell you.

Ask what changed against what is written

Look at the infrastructure changes merged in the last two weeks, and read
our runbooks.

Tell me where a runbook now contradicts reality:
  which runbook, which step
  what the runbook says
  what the change suggests is now true
  how confident you are

Only report contradictions, not gaps. A runbook that does not mention a new
service is incomplete; one that tells you to restart a service that no
longer exists is dangerous. I want the second.

If nothing contradicts, say so.

The distinction between incomplete and wrong is the whole prioritisation. Every runbook is incomplete, permanently, and a report of gaps is infinite and unactionable. A runbook that gives an instruction which no longer works is a specific, finite, fixable problem, and it is the one that costs you at 3am.

Draft the edit, do not apply it

For each contradiction, draft the corrected step. Show the current text and
the proposed text side by side.

Do not edit the runbook. If you are not confident what the new behaviour
is, say so and leave it for a human.

A runbook edited automatically is a runbook nobody has verified, and it will be followed literally under pressure by someone who assumes a person wrote it.

Run it after changes, not on a calendar

A fortnightly schedule is fine, but the better trigger is after a significant infrastructure change. That is when the contradiction is introduced and when whoever made it still remembers why.

Fix the confident ones, investigate the rest

The confidence field is doing real work. High-confidence contradictions are usually a renamed service or a removed step, and take a minute. Low-confidence ones are usually where the runbook encoded a judgement, and those want a conversation.

What good looks like

Runbooks that have not been actively wrong for a quarter, and a short list after each infrastructure change of what needs a sentence rewritten.

The measure is the next incident: did anyone following the runbook hit a step that no longer applies?