Engineering & IT

Five errors worth fixing, every morning

Overnight noise grouped into what is new or growing, rather than what is loudest.

Time
14 minutes
You work in
Scheduled Jobs
Connect first
Sentry

Before you start

  • Sentry connected
  • An employee you have hired

Error tracking has a volume problem, not a detection problem. Everything is caught, nothing is triaged, and the dashboard becomes a wall people stop looking at. The loudest error is rarely the most important one, and it is always at the top.

Connect Sentry

From Integrations, search for Sentry and connect it. Whatever you use is fine as long as it groups errors rather than logging lines.

Ask for new or growing, not for most frequent

This is the whole idea. Frequency is a property of how much traffic hits a code path, not of how badly something is broken.

Prompt
Every weekday at 7am, review errors from the last 24 hours.

Give me at most five, chosen on this basis:
  anything that appeared for the first time
  anything whose rate is climbing against its own baseline
  anything affecting more than a handful of distinct users

For each: what it is in one sentence, where it happens, how many users,
whether it is new or growing, and the first line of the stack.

Explicitly ignore anything that is high volume but flat. Those are known
noise and I am not asking about them today.

If nothing meets the bar, say "nothing new" and stop.

"High volume but flat" is the category that ruins these reports. Every codebase has two or three errors that fire constantly and harm nobody, and they will dominate any list sorted by count. Excluding them by name is what makes the remaining five worth reading.

Read it for a week before acting on it

You are calibrating, not fixing. After five days you will know which of your persistent errors need adding to the ignore list, and whether "growing" is picking up real trends or just Monday traffic.

That is a prompt edit each time, and the report is noticeably better by the end of the week.

Turn the survivors into work

Once the five are trustworthy, the natural extension is to open an issue for anything genuinely new, rather than just reporting it.

Do this only after the calibration week. An employee opening tickets from an uncalibrated report will bury your backlog in noise, and that costs more goodwill than the reports save.

Say the quiet part in the prompt

Add this line once you are running for real:

Prompt
If the same error appears in this report three days running and nobody has
acted on it, say so and stop including it. Tell me it is being ignored
rather than reporting it a fourth time.

An error nobody fixes after three mentions is not going to be fixed by a fourth. Naming that is more useful than repeating it.

What good looks like

Five things at 7am, most mornings fewer, and the number is small enough that someone actually reads it before standup.

The failure mode to watch for is a report that is always exactly five items long. That means the bar is being filled rather than met, and the fix is to insist that fewer is allowed.