Overview / Alerting policy
Alerting policy
An alert is a request for somebody's attention
Attention is the scarcest thing an operations team has, and every alert spends some of it. That single idea decides almost every question on this page: what to alert on, who to tell, and when to delete an alert that has stopped earning its keep.
First principle
Alert on symptoms, not on causes
A symptom is something a user would recognise: orders are failing, the booking confirmation is taking far longer than usual, a queue of claims is growing and not draining. A cause is a fact about a machine: processor use is high, a disk is filling, a service restarted.
Alerting on causes produces two failures at once. It fires when nothing is wrong — high processor use during a nightly batch is the system working correctly — and it stays silent when something is very wrong for a reason nobody thought to write a rule for. The set of possible causes is unbounded. The set of symptoms your business cares about is short, and you can list it in an afternoon.
Causes still deserve to be measured, in detail, because they are what you look at once an alert has fired. The distinction is not between important and unimportant data. It is between data that should wake somebody and data that should be waiting for them when they arrive.
The cost of a false alarm, measured in trust
The obvious cost of a false alarm is the time it wastes. That is the smaller half. The real cost is what it does to how the next alert is read.
Every alert that turns out to be nothing teaches the team a small lesson: this signal is not reliable. Nobody decides to ignore it, and there is rarely a meeting about it. The response time just gets a little longer, the checking a little less thorough, until one day the alert that mattered is the one that sat for forty minutes because everybody assumed it was the usual noise. Trust is spent gradually and it cannot be topped up by explaining that this time it is serious.
Which means the quality bar for anything that interrupts a person is much higher than it feels while you are writing the rule. Our recommended test is uncomfortable on purpose: would you be comfortable waking a colleague at three in the morning for this, every single time it fires, for the next six months? If the honest answer is no, it is not an alert. It is a dashboard, or a ticket, or a line in a weekly review — all of which are respectable destinations.
A second, quieter cost is worth naming. Noisy alerting pushes teams towards suppression: a threshold nudged up, a rule muted “temporarily”, a filter that drops a whole class of events. Each step is reasonable and the accumulation is a system that no longer reports what it knows.
Who is woken, and for what
Deciding to wake somebody is a decision about a person's night, not a technical setting, so it belongs to a named human policy rather than to whoever last edited the configuration. Three questions settle almost every case.
| Question | If yes | If no |
|---|---|---|
| Is a real business transaction failing or badly degraded right now? | Continue to the next question. | Not an alert. Record it and look at it in working hours. |
| Is there an action a person could take in the next few minutes that would help? | Continue. | Waking somebody buys nothing. Capture the evidence for later analysis instead. |
| Is the person you would wake the one who can take that action? | Send it, with the transaction name and the hop in the first line. | Fix the routing first. An alert delivered to somebody who has to forward it has wasted the very time it was trying to save. |
What the message contains matters as much as who receives it. A person woken at night should learn, in one line and before opening anything, which business transaction is affected, roughly how much of it is affected, which hop the evidence points at, and when it started. Everything else can be a link. An alert that says only that a threshold was crossed makes the reader do the diagnosis from a standing start, half asleep.
How to retire an alert nobody acts on
Alerts accumulate. Each one was added for a good reason, usually after an incident, and almost nothing in the normal run of work ever removes one. The result is a configuration nobody fully understands, where a meaningful signal arrives alongside a dozen that everyone has learned to skip.
Retirement needs to be a routine, not an act of courage. The routine that works is built on a single piece of record-keeping: for every alert that fires, note what was done. Not what caused it — what a person actually did. Four answers cover nearly everything: something was changed, it recovered on its own, it was a false alarm, or nothing was done.
- Review on a schedule, monthly or quarterly, with the firing record in front of you. Without a fixed date this never happens.
- An alert that has fired repeatedly with “nothing was done” every time is not protecting you. Either the condition is harmless, or the action is impossible — and both mean it should stop interrupting people.
- Downgrade before deleting. Move it from waking somebody to a ticket, or from a ticket to a dashboard. Most of the benefit of deletion, and it is reversible.
- Delete rather than mute. A muted rule is a landmine: it is still in the configuration, still looks like coverage on paper, and still fires the day somebody clears the suppression list.
- Record why it went. One line in the same place the alert was defined, so the next person to have the same idea can read what happened last time.
- Retire the alert, keep the measurement. The underlying data costs almost nothing to continue collecting, and it is what makes the next investigation quick.
The uncomfortable version of this rule: an alert nobody acts on is worse than no alert, because it costs attention and returns nothing while creating the impression that the area is covered. Deleting it makes the gap visible, which is the first step to filling it properly.
Questions that come up in review
Should an alert fire on a single slow transaction?
Almost never. One slow transaction is a fact about one transaction, and in any real system a few of them are slow at all times for reasons that are nobody's fault. Alert on a proportion over a window — a share of transactions of one named kind exceeding a duration for a sustained period — so that a single unlucky event cannot spend somebody's night.
What about the opposite: a transaction that stopped arriving?
This is the alert teams most often lack, and it is frequently the most valuable one. A transaction type that normally runs steadily and has now run zero times is a strong signal, and no error-rate rule will ever notice it because there are no errors. It does need a seasonal baseline, or it will fire every weekend.
How many alerts should one team have?
Few enough that a new joiner can read all of them in one sitting and understand what each is protecting. If that is impossible, the set has stopped being a policy and has become sediment.
Do you page on a warning?
We would rather not have a warning severity at all. In practice a two-level scheme — interrupt a person now, or put it in the queue for working hours — forces the useful conversation. A middle level tends to collect everything nobody wanted to argue about.
What happens when an alert fires during a deployment?
It should say so. An alert that knows a release started a few minutes ago, and says it in the first line, saves the reader the single most common diagnostic step. That is a reporting improvement, not a reason to suppress alerts during releases — the deployment window is exactly when you most want to be told.
Bring your noisiest alert to the demo
The fastest way to judge whether any of this is useful is to look at a rule that already fires too often, and work out together what it should have been measuring.