Overview / Service baselines
Service baselines
Normal is a local fact about your system
A baseline is the answer to “compared with what?”. Without one, every measurement is just a number, and every conversation about performance collapses into whoever remembers last month most confidently.
Where a baseline comes from
Your own system, not a published figure
Industry figures for how fast a transaction “should” be are almost always useless, and not because they are dishonest. They are measurements of a different system: a different data volume, a different hardware generation, a different mix of customers, a different set of features switched on, a network with different distances in it.
The only baseline that can catch a regression is one built from your own transactions on your own installation. It has a property no external figure can have: it already contains all the things about your system that are permanently true, so anything that departs from it is genuinely new.
This has an uncomfortable consequence worth stating. A brand-new installation has no baseline, and it cannot have one until it has watched a representative stretch of ordinary operation. In the first days, a monitoring tool can tell you what is happening and cannot yet tell you whether it is unusual. Anyone promising otherwise is comparing you to somebody else.
Percentiles against averages
A percentile is a plain idea: sort every transaction by duration and read off the value at a given position in the list. The median is the value halfway along, so half of transactions were faster and half slower. The ninety-fifth percentile is the value at the point where nineteen in twenty were faster. That is the whole definition, and its usefulness comes from what it refuses to do: it never invents a value nobody experienced, because it always reports a duration that some real transaction actually had.
Averages fail on this exact point, as set out on the overview. A mean can move because of a handful of extreme values, or fail to move while a substantial minority of your users have a bad time. Percentiles let you say something specific instead: not “it is slower” but “the slowest one in twenty got worse while the median did not move” — which is a completely different bug.
Two practical warnings. First, high percentiles need volume behind them: a percentile deep in the tail computed from a handful of transactions is describing individual events, not a pattern. Second, and this one is a genuine mathematical fact rather than a preference: you cannot average percentiles. The ninety-fifth percentile of a combined set is not the average of the ninety-fifth percentiles of its parts, and combining them that way will produce a number that looks authoritative and corresponds to nothing. Percentiles have to be computed from the underlying distribution, which is why our storage keeps the shape of the data rather than only the summary figures.
Seasonality
Systems that serve a business inherit the business's calendar. Traffic follows the working day, the working week, the payroll cycle, the end of the month, the end of the quarter, the school term, the sales campaign somebody in marketing scheduled without telling you. None of this is noise. It is structure, and a baseline that ignores it will be wrong in a predictable daily rhythm.
The practical rule is to compare like with like: this hour against the same hour on the same weekday, rather than against the hour that has just passed. A transaction mix at nine in the morning is genuinely different from the one at three in the afternoon, so comparing the two tells you about the clock rather than about the system.
It also means a baseline needs enough history to have seen a full cycle of whatever periods matter to you. If your business has a strong monthly shape, a baseline built from two weeks will treat every month-end as an emergency.
How a baseline goes stale
A baseline is a statement about a specific version of a system. The moment you deploy, that statement may no longer be true, and the tool has no way of knowing whether the change it can see is a regression to be alarmed about or an intended consequence of the release.
Three things routinely invalidate a baseline, and they are worth separating because the correct response differs:
- A deliberate change in the code
- A new feature adds a hop; a query is rewritten; a cache is introduced. The old baseline is now describing software that no longer exists. The response is to re-baseline on purpose, from a stated point, and to keep the old one for comparison rather than deleting it.
- A change in the data or the traffic
- A table has grown past the point where an index helps, or a large customer has been added. Nothing in the code changed, but the system's normal genuinely has. This is the case where a slowly drifting baseline is dangerous: if the baseline follows the drift, it will quietly accept a system that is degrading week by week.
- A change in the environment
- New hardware, a different network path, a neighbour on the same host. The response is usually to re-baseline, but only after checking that the change was intended — an unexplained environmental shift is itself the finding.
Because of the second case, we do not let a baseline drift silently. A baseline here has a date, a version it belongs to, and a visible record of when it was last recalculated. If the current behaviour and the baseline have separated, that is shown as a difference to be explained, not smoothed away by a moving window that always agrees with the present.
Signature tool
Spend the budget before you build the system
A target end-to-end time is not a wish. It is a quantity that has to be divided among the hops that will consume it, and the division is where the argument belongs — on a whiteboard, before anything is written.
Latency budget allocator
Set the end-to-end time you are designing for, then hand it out across the six hops. Every figure below is arithmetic on what you type — nothing is sent anywhere, and the starting numbers are only there to be replaced with yours.
- Target
- —
- Allocated
- —
- Remaining or over
- —
- Largest single hop
- —
The allocator does arithmetic and nothing else: it adds up what you type, subtracts it from your target, works out each hop's share of the allocation, and names the largest. It holds nothing, sends nothing, and has no opinion about your architecture. Two things it deliberately does not model: hops that run at the same time rather than one after another, and the overhead the instrumentation itself adds. Leave a little of the budget unallocated for both.
Questions that come up when setting a baseline
How long should a baseline window be?
Long enough to contain a full cycle of every period that genuinely affects you, and short enough that it still describes the current system. Those two pull against each other, which is why the window is a decision rather than a default. Write down which periods you believe matter; the window follows from that list.
Should the baseline include incidents?
No, if you can identify them, and yes by default if you cannot. A baseline that includes last month's outage has learned that being broken is normal. Excluding periods needs a record of what was excluded and why, otherwise it becomes a quiet way of making the numbers look better.
One baseline per transaction, or one for the service?
Per transaction type, always. A service that handles a fast lookup and a heavy report has no meaningful single normal, and a combined baseline will hide a doubling of the fast path under the noise of the slow one.
What about a transaction that is rare?
Treat it as a counted event rather than a distribution. For something that runs a few times a day, individual durations and a record of whether it finished are more informative than any percentile you could compute from so few points.
Does a baseline replace a target?
No, and confusing the two causes real damage. A baseline says what your system does. A target says what it ought to do. A system can sit comfortably on its baseline while being far too slow for the people using it, which is precisely the situation the budget allocator above is meant to expose.
A baseline is only worth having if something happens when reality departs from it. That is the subject of the alerting policy.