Free lesson · SRE and CloudOps on AWS

SLIs, SLOs, and error budgets with CloudWatch

Easy9 min readawssrereliability

The Three Terms

The error budget is the point. An SLO without one is a number in a document; an error budget is a quantity you spend, and spending it has consequences.

Choosing SLIs

An SLI must measure something a user experiences. CPU utilisation is not an SLI. Neither is instance count, queue depth, or cache hit rate - those are useful metrics and they are not measures of whether the service is working for someone.

Two or three SLIs per service is right. Availability and latency covers most request-driven services; a pipeline usually needs freshness and correctness.

Tip: The most common SLI mistake is defining "valid requests" carelessly, and it determines whether the number means anything.

A 4xx caused by a malformed client request is not your service failing - including it means your SLI degrades when a client ships a bug, and you burn error budget for something you cannot fix. A 429 from your own throttling is more ambiguous: the user experienced a failure, and you chose it deliberately.

Write the definition down explicitly: which status codes count as failures, which requests are excluded, and what happens to requests that never reached your service at all. Two engineers computing the same SLI should get the same number, and without a written definition they will not.

Where You Measure Changes The Answer

graph LR
    U[User] --> I{Internet}
    I --> CF[CloudFront]
    CF --> ALB[ALB]
    ALB --> A[Application]
    I -.->|**never arrived** —<br/>invisible to every<br/>server-side metric| X[DNS failure,<br/>network, TLS]
    CF -.->|CloudFront metrics| M1[Closest to the user<br/>server-side]
    ALB -.->|ALB metrics| M2[Misses anything<br/>before the ALB]
    A -.->|application metrics| M3[Misses ALB 5xx,<br/>target health failures]
    U -.->|RUM / synthetics| M4[**Closest to truth**]

This diagram shows measurement points for a service level indicator. A user request crosses the internet to CloudFront, then an Application Load Balancer, then the application. Requests that never arrive due to DNS failure, network problems, or TLS issues are invisible to every server-side metric. CloudFront metrics are the closest server-side measurement to the user. ALB metrics miss anything failing before the load balancer. Application metrics additionally miss load balancer 5xx errors and target health failures. Real user monitoring and synthetics measure closest to the truth.

Warning: Application-level metrics will report better availability than your users experience, and the gap is invisible.

A request that failed at DNS, timed out before reaching the load balancer, or was rejected by the load balancer because no targets were healthy never touched your application and does not appear in its metrics. During a total outage, application error rate can look perfect - because nothing arrived to fail.

Measure as close to the user as you can. ALB metrics are a reasonable practical floor for most services; CloudFront metrics are better; RUM and synthetics are closest and cover the paths nothing server-side can see.

The concrete case worth remembering: an ALB with zero healthy targets returns 5xx and your application logs nothing at all. Application-side SLIs report 100% availability during a complete outage.

Choosing The SLO

The number is a business decision with an engineering cost, and each nine costs disproportionately more than the last.

Important: Look at the 99.99% row and ask what you could actually do in four minutes. Detection alone is frequently longer than that. A four-nines SLO means most incidents must be handled without human involvement - automatic failover, automatic rollback, and no manual step in the recovery path.

That is achievable and it is expensive. Choosing 99.99% without the automation to support it produces an SLO that is missed every month, at which point everyone stops looking at it.

Two better inputs than aspiration. What does the business actually require - is there a contractual SLA, and what does an hour of downtime cost? And what are you currently achieving - measure for a month before setting a target, because an SLO below current performance is meaningless and one far above it is unachievable.

Never set 100%. It leaves no error budget, which means no room to deploy, and it is not attainable in any case since your dependencies are not 100%.

Implementing In CloudWatch

The SLI as a ratio, built with metric math.

For an ALB-fronted service, good requests are total requests minus 5xx errors - from RequestCount, HTTPCode_ELB_5XX_Count, and HTTPCode_Target_5XX_Count. Metric math computes the ratio, and an alarm on the result is availability alarming rather than error-count alarming.

For a latency SLI, the cleanest implementation emits a custom metric counting requests under the threshold - via EMF, so there is no API call in the request path (Course 1 Module 08 Topic 1). The percentile statistic on a latency metric tells you the p99 value; it does not directly give you the proportion under a threshold, which is what the SLI needs.

CloudWatch Application Signals provides managed SLOs - define the objective, and it computes attainment and error budget. Verify current capability, and where it fits it removes the metric math work entirely.

Burn Rate Alerting

The technique that makes error budgets operational, and the reason it exists is worth stating.

Alerting when the SLO is breached is too late - the budget is already spent. Alerting on any error rate above zero is noise. Burn rate alerting solves both.

Burn rate is how fast you are consuming the budget relative to the window. A burn rate of 1 exhausts the budget exactly at the end of the period. A burn rate of 14.4 exhausts it in about 2% of the period.

Multi-window, multi-burn-rate is the standard approach:

Tip: The two windows exist to catch two different failures, and using one window catches only one of them.

A short window alone produces false positives - a brief spike burns budget fast and recovers, and paging on it wakes someone for something already over.

A long window alone misses the slow burn - a persistent 0.3% error rate never trips a short-window threshold and quietly consumes the entire month's budget.

Requiring both a short and a long window to breach removes the false positives while retaining fast detection: the short window provides speed, the long window provides confirmation that it is sustained.

Implement it with a composite alarm requiring both child alarms - Course 1 Module 08 Topic 4.

The Error Budget Policy

The part that makes any of this matter, and the part most organisations skip.

Important: An error budget with no agreed consequence is a metric, not a mechanism. The question - what happens when the budget is exhausted? - must be answered in advance, in writing, with agreement from engineering and product leadership.

A workable policy:

Budget state	Response
Healthy	Ship normally. Take reliability risks deliberately.
Under 25% remaining	Reliability work prioritised alongside features
Exhausted	Feature work pauses; reliability work only, until recovered
Repeatedly exhausted	The SLO or the architecture is wrong - revisit both

The value is that the argument happens once, calmly, rather than during every incident. Without it, reliability competes with features on opinion each time, and it loses each time - which is the situation the error budget exists to fix.

The policy must be agreed by whoever can stop feature work. An error budget policy the engineering team wrote and nobody else accepted will not survive its first invocation.

A budget consistently unspent is also a signal - the SLO may be too lenient, or you are over-investing in reliability relative to what users need. Both directions are information.

Reporting And Review

Monthly, with a fixed shape: attainment against each SLO, budget consumed and remaining, the incidents that consumed it, and any policy invocation.

Show it to the people who set priorities. An SLO report circulating only among engineers cannot influence the trade-offs it exists to inform.

Revisit the SLOs themselves quarterly. A service whose SLO has never been missed may be over-targeted; one missed every month is either mis-set or the architecture cannot support it.

Common Failure Modes

Warning: Measuring at the application. Reports availability during a total outage.

"Valid requests" undefined. Two people compute different numbers.

SLO set aspirationally. Missed monthly, then ignored.

100% as the target. No budget, no room to deploy.

No error budget policy. A metric, not a mechanism.

Policy not agreed by product leadership. Does not survive first invocation.

Single-window burn rate alerting. False positives or slow-burn blindness.

Too many SLIs. Nobody knows which matters.

CPU or queue depth as an SLI. Not user-experienced.

SLOs never reviewed. Targets drift from reality.

Reports seen only by engineers.

Budget consistently unspent and never questioned.

Why This Matters In An Interview

Interview: SLO questions test whether reliability is being managed or described.

What interviewers are listening for:

• Measurement point affecting the number, with the zero-healthy-targets case.
• Multi-window multi-burn-rate, and why both windows.
• The error budget policy as the mechanism, agreed in advance.
• Willingness to say 99.99% is expensive and often unnecessary.

Common follow-ups:

• "Define an SLI for a checkout service." → The proportion of valid checkout requests served successfully within a latency threshold, measured at the load balancer or edge. Then define valid - excluding client-caused 4xx - and explain why application-level measurement would over-report.
• "Why not measure availability in the application?" → It misses everything that failed before reaching it. With zero healthy targets the ALB returns 5xx and the application logs nothing, so the SLI reports 100% during a complete outage.
• "Should we target 99.99%?" → That is about four minutes a month, which means most incidents must resolve without human involvement. Achievable and expensive. What does the business require, and what are you currently achieving?
• "What happens when the error budget is exhausted?" → Whatever the policy says, agreed in advance with product leadership. Typically feature work pauses until it recovers. Without a policy the budget is a metric rather than a mechanism.

Practice

Exercise: Starter. For one service, measure current availability at both the application and the load balancer for a month. The difference is what application-level measurement hides.

Intermediate. Implement an availability SLI with CloudWatch metric math and a multi-window burn rate composite alarm. Verify it does not fire on a brief spike and does fire on a sustained low error rate.

Advanced. Write an error budget policy, take it to product leadership, and get it agreed. The document is the easy part.

Related Topics

• Topic 1 - Why the error budget provides SRE's authority
• Module 02 Topic 2 - Burn rate alerting within the alerting design
• Module 05 Topic 1 - Incidents as budget consumption
• Course 1 Module 08 Topics 1, 4 - Metrics, metric math, composite alarms
• Course 3 Module 05 Topic 2 - The platform-provided SLO framework

Back to SRE and CloudOps on AWS